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1. INTRODUCTION 



This methodology report provides technical information about the development, design, and 
conduct of the third grade data' collection of the Early Childhood Longitudinal Study, Kindergarten Class 
of 1998-99 (ECLS-K). It begins with an overview of the ECLS-K study. Subsequent chapters provide 
information on the development of the instruments, sample design, data collection methods, data 
preparation and editing, response rates, and weighting and variance estimation. 

The ECLS-K focuses on children’s early school experiences beginning with kindergarten. It 
is a multisource, multimethod study that includes interviews with parents; the collection of data from 
principals, teachers, and student records abstracts; and direct child assessments. The ECLS-K was 
developed under the sponsorship of the U.S. Department of Education, National Center for Education 
Statistics (NCES). Westat is conducting this study with assistance provided by Educational Testing 
Service (ETS) in Princeton, New Jersey. The Survey Research Center and the School of Education at the 
University of Michigan assisted Westat in conducting the base year and first grade study. 

The ECLS-K follows a nationally representative cohort of children from kindergarten 
through fifth grade. The base year data were collected in the fall and spring of the 1 998-99 school year 
when the sampled children were in kindergarten. A total of 21,260 kindergartners throughout the nation 
participated. 



Two more waves of data were collected in the fall and spring of the 1 999-2000 school year 
when most, but not all, of the base year children were in first grade.^ The fall-first grade data collection 
was limited to a 30 percent subsample of schools. Approximately 27 percent of the base year students 
who were eligible to participate in year 2 attended the 30 percent subsample of schools (see exhibit 1-1). 
The fall-first grade data collection was a design enhancement to enable researchers to measure the extent 
of summer learning loss and the factors that contributed to such loss and to better disentangle school and 
home effects on children’s learning. The spring-first grade data collection, on the full sample, was part of 
the original study design and can be used to measure annual school progress and to describe the first 
grade learning environment of children in the study. All children assessed during the base year were 



’ The term “third grade” is used throughout this document to refer to the data collection that took place in the 2001-2002 school year, at which 
time most of the sampled children — but not all of them — were in third grade. 

^ Though the majority of base year children were in first grade during the 1999-2000 school year, about 5 percent of the sampled children were 
retained in kindergarten and a handful of others were in second grade during the 1999-2000 school year. 
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eligible to be assessed in the spring-first grade data collection regardless of whether they repeated 
kindergarten, were promoted to first grade, or were promoted to second grade. In addition, children who 
were not in kindergarten in the United States during the 1998-99 school year and, therefore, did not have 
a chance to be selected to participate in the base year of the ECLS-K were added to the spring-first grade 
sample.^ Such children included immigrants to the United States who arrived after fall 1998 sampling, 
children living abroad during the 1998-99 school year, children who were in first grade in 1998-99 and 
repeated it in 1999-2000, and children who did not attend kindergarten. Their addition allows researchers 
to make estimates for all first graders in the United States rather than just for those who attended 
kindergarten in the United States in the previous year. 



Exhibit 1-1. ECES-K waves of data collection: School years 1998-99, 1999-2000, 2001-02, and 
2003-04 



Data collection 


Date of collection 


Sample 


F all-kindergarten 


Fall 1998 


Full sample 


Spring-kindergarten 


Spring 1999 


Full sample 


Fall-first grade 


Fall 1999 


30 percent subsample' 


Spring-first grade 


Spring 2000 


Full sample 


Spring-third grade 


Spring 2002 


Full sample 


Spring-fifth grade 


Spring 2004 


Full sample 



NOTE: See section 1.3 for a description of the study components. More information is provided in the Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 (ECLS-K) User’s Manual for the ECLS-K Third Grade Public-Use Data File and Electronic Code Book, (NCES 
2004—001; U.S. Department of Education, National Center for Education Statistics, 2004). 

' Fall data collection consisted of a 30 percent sample of schools containing approximately 27 percent of the base year students eligible to 
participate in year 2. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99. 



A fifth wave of data was collected in the spring of the 2001-02 school year when most, but 
not all, of the sampled children were in third grade. Approximately 89 percent of the children interviewed 
were in third grade during the 2001-02 school year, 9 percent were in second grade, and less than 1 
percent were in fourth grade or higher. In addition to the school, teacher, parent, and child assessment 
data collection components, children were asked to complete a short self-description questionnaire, which 
included questions about what they were like and their likes and dislikes. The spring-third grade data 
collection can be used to measure school progress and to describe the third grade learning environment of 
children in the study. 

^ Their addition is referred to as “freshening” the sample. See chapter 3 for more detail on the freshening process. 
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The sample of children in the third grade round of data collection of the ECLS-K represents 
the cohort of children who were in kindergarten in 1998-99 or in first grade in 1999-2000. Since the 
sample of children fielded in 2001-02 was not freshened with third graders who did not have a chance to 
be sampled in kindergarten or first grade (for example, because they were out of the country during their 
kindergarten and first grade years), this sample of children does not represent all third graders in 2001-02. 
The vast majority of children in third grade in the 2001-02 school year are members of the cohort. 
However, third graders who repeated second or third grade and recent immigrants were not covered. Data 
were collected from teachers and schools to provide important contextual information about the 
environment for the sampled children. The teachers and schools are not representative of third grade 
teachers and schools in the country in 2001-02 but representative only of teachers of the cohort of 
children sampled in kindergarten in 1998-99 and freshened in spring- first grade in 2000. For this reason, 
the only weights produced from the study are for making statements about children, including statements 
about the teachers and schools of those children. 

The final wave of data collection as currently planned is scheduled for spring 2004 when 
most of the sampled children will be in the fifth grade. 

The ECLS-K has several major objectives and numerous potential applications. The ECLS- 
K combines (1) a study of achievement in the elementary years; (2) an assessment of the developmental 
status of children in the United States at the start of their formal schooling and at key points during the 
elementary school years; (3) cross-sectional studies of the nature and quality of kindergarten programs in 
the United States; and (4) a study of the relationship of family, preschool, and school experiences to 
children’s developmental status at school entry and their progress during the kindergarten and early 
elementary school years. 

The ECLS-K is part of a longitudinal studies program comprising two cohorts — a 
kindergarten cohort and a birth cohort. The birth cohort (ECLS-B) is following a national sample of 
children born in the year 2001 from birth through first grade. The ECLS-B focuses on the characteristics 
of children and their families that influence children’s first experiences with the demands of formal 
school, as well as children’s early health care and in- and out-of-home experiences. Together these 
cohorts will provide the depth and breadth of data required to more fully describe and understand 
children’s health and early learning, development, and education experiences. 
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The ECLS-K has both descriptive and analytic purposes. It provides descriptive data on 
children’s status at school entry, their transition into school, and their progress through fifth grade. The 
ECLS-K also provides a rich data set that enables researchers to analyze how a wide range of family, 
school, community, and individual variables affect children’s early success in school; explore school 
readiness and the relationship between the kindergarten experience and later elementary school 
performance; and record children’s cognitive and academic growth as they move through elementary 
school. 



1.1 Background 

National policymakers and the public at large have increasingly recognized that the 
prosperity of the United States depends on the successful functioning of the American education system. 
There is also growing awareness that school reform efforts cannot focus solely on the secondary and 
postsecondary years but must pay attention to the elementary and preschool years as well. Increased 
policy interest in the early grades and the early childhood period is reflected in an intensified recent 
national policy aimed at ensuring that children are capable of reading by the third grade, providing 
college student and adult volunteer tutors for children who are having difficulty learning to read, and 
preparing children to succeed in school with improved Head Start and early childhood development 
programs. 



Efforts to expand and improve early education will benefit from insights gained through 
analyses of data from the large scale, nationally representative ECLS-K data and the study’s longitudinal 
design. The ECLS-K database contains information about the types of school programs in which children 
participated, the services they received, and repeated measures of the children’s cognitive skills and 
knowledge. The ECLS-K database also contains measures of children’s physical health and growth, social 
development, and emotional well-being, along with information on family background and the 
educational quality of their home environments. 

As a study of early achievement, the ECLS-K allows researchers to examine how children’s 
progress is affected by such factors as placement in high or low ability groups, receipt of special services 
or remedial instruction, grade retention, and frequent changes in schools attended because of family 
moves. Data on these early school experiences are collected as they occur, with the exception of their 
experiences before kindergarten, which were collected retrospectively. This produces a more accurate 
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measurement of these antecedent factors and enables stronger causal inferences to be made about their 
relationship to later academic progress. 

The ECLS-K enables educational researchers and policy analysts to use a variety of 
perspectives on early childhood education, using techniques such as multilevel modeling to study how 
school and classroom factors affect the progress of individual children. The data collected enable analysts 
to examine how children’s status at school entry and performance in school are determined by an 
interaction of child characteristics and school and family environments. 

Data collected during the kindergarten year serve as baseline measures to examine how 
schooling shapes later individual development and achievement. The longitudinal nature of the study 
enables researchers to study children’s cognitive, social, and emotional growth and to relate trajectories of 
change to variations in children’s experiences in kindergarten and the early grades. 

The spring-third grade data collection can be used to describe the diversity of the study 
children and the classrooms and schools they attend. It can also be used to study children’s academic 
gains in the years following kindergarten and first grade. The ECLS-K sample includes substantial 
numbers of children from various minority groups. Thus, the ECLS-K data present many possibilities for 
studying cultural and ethnic differences in the educational preferences and literacy practices of families, 
the developmental patterns and learning styles of children, and the educational resources and 
opportunities that different groups are afforded in the United States. 



1.2 Conceptual Model 

The design of the ECLS-K has been guided by a framework of children’s development and 
schooling that emphasizes the interrelationships between the child and family, the child and school, the 
family and school, and the family, school, and community. The ECLS-K recognizes the importance of 
factors that represent the child’s health status and socioemotional and intellectual development and 
incorporates factors from the child’s family, community, and school-classroom environments. The ECLS- 
K conceptual model is depicted in exhibit 1-2. The study has paid particular attention to the role that 
parents and families play in helping children adjust to formal school and in supporting their education 
through the primary grades. It has also gathered information on how schools prepare for and respond to 
the diverse backgrounds and experiences of the children and families they serve. 
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Exhibit 1-2. ECLS-K conceptual model: School years 1998-2004 




SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99. 



1.3 Study Components 



The emphasis placed on measuring children’s environments and development broadly has 
critical implications for the design of the ECES-K. The design of the study includes the collection of data 
from the child, the child’s parents/guardians, teachers, and schools. 



■ Children participate in various activities to measure the extent to which they exhibit 
those abilities and skills deemed important for success in school. They are asked to 
participate in activities designed to measure important cognitive (general knowledge, 
science, literacy, and quantitative) skills and noncognitive (fine and gross motor 
coordination [in kindergarten] and socioemotional development) skills and 
knowledge. All measures of a child’s cognitive skills are obtained through an untimed 
one-on-one assessment of the child. Beginning with the third grade data collection, 
children report on their own perceptions of their abilities and achievement as well as 
their interest in and enjoyment of reading, math, and other school subjects. Children 
are assessed in each round of data collection. Children’s height and weight are also 
measured in each round of data collection. 

■ Parents/guardians are an important source of information about the families of the 
children selected for the study and about themselves. Parents provide information 
about children’s development at school entry and at various points in time and about 
their experiences both with family members and others. Information is collected from 
parents each time children are assessed using computer-assisted interviews (CAIs). 
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■ Teachers, like parents, represent a valuable source of information on themselves, the 
children in their classrooms, and the children’s learning environment (i.e., the 
classroom). Teachers are not only asked to provide information about their own 
backgrounds, teaching practices, and experience, they are also called on to provide 
information on the classroom setting for the sampled children they teach and to 
evaluate each sampled child on a number of critical cognitive and noncognitive 
dimensions. Special education teachers and service providers of sampled children with 
disabilities are also asked to provide information on the nature and types of services 
provided to the child. With the exception of the fall-first grade data collection, 
teachers complete self-administered questionnaires each time children are assessed. 

■ School administrators, or their designees, are asked to provide information on the 
physical, organizational, and fiscal characteristics of their schools, and on the schools’ 
learning environment and programs. Special attention is paid to the instructional 
philosophy of the school and its expectations for students. Information is collected 
from school administrators via self-administered questionnaires during each spring 
data collection. With the exception of first grade, the administrators of all ECLS-K 
schools use a single questionnaire. In the first grade two different questionnaires were 
used, one for original sampled schools and one for new or transfer schools. More 
information can be found in the Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 (ECLS-K), User’s Manual for the ECLS-K First Grade Public-Use 
Data Files and Electronic Code Book (NCES 2002-135; U.S. Department of 
Education, National Center for Education Statistics, 2002b). 

■ School office staff are asked to complete a student records abstract form and a school 
fact sheet. The student records abstract form includes questions about an individual 
child’s enrollment and attendance at the school, transfer to another school (if 
applicable), and verifies whether the child has an Individual Education Plan (lEP) on 
record. A student records abstract form is completed for each child in the study during 
each spring data collection. 

During the third grade data collection, school office staff are also asked to complete a 
school fact sheet. This form supplements the school administrator questionnaire with 
basic information about the school, including grade level, school type (public or 
private), length of school year, and attendance recordkeeping practices. This school 
fact sheet is only filled out once for each school in the study. Prior to the third grade 
data collection, the questions were part of the school administrator questionnaire. 

Exhibit 1 -3 summarizes the instruments that were used in each of the data collection periods 
from kindergarten through spring-third grade. Exhibit 1 -4 provides additional detail about the direct child 
assessments conducted during each of the data collection periods. Separate psychometric reports have 
been prepared to describe the design and development of the kindergarten through first grade and third 
grade assessment batteries. For detailed information about the child assessments, including their 
psychometric properties, see Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 
(ECLS-K), Psychometric Report for Kindergarten Through First Grade, NCES 2002-05 (U.S. 
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Department of Education, National Center for Education Statistics, 2002a) and Early Childhood 
Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), Psychometric Report for the Third Grade 
(U.S. Department of Education, National Center for Education Statistics (NCES 2004-087, forthcoming). 

Exhibit 1-3. Instruments used in the ECES-K, by round of data collection: School years 1998-99, 
1999-2000, and 2001-02 



Instruments 


1998- 

school 


1999 

year 


1999-2000 
school year 


2001-02 
school year 


Fall- 

kindergarten 


Spring- 

kindergarten 


Fall- 
first grade' 


Spring- 
first grade 


Spring- 
third grade 


Parent interview 


X 


X 


X 


X 


X 


Child assessments 


X 


X 


X 


X 


X 


Teacher questionnaire part A 


X 


X 


X 


X^ 


X 


Teacher questionnaire part B 


X 


X 


X 


x^ 


X 


Teacher questionnaire part C 


X 


X 


X 


x^ 


X 


Special education teacher 












questionnaire part A 




X 




X 


X 


Special education teacher 












questionnaire part B 




X 




X 


X 


Adaptive Behavior Scale 




X 




X 




Self-Description Questionnaire 










X 


School administrator questionnaire 




X 




x^ 


X 


Student records abstract 




X 




X 


X 


School fact sheet 










X 


School facilities checklist 




X 




X 


X 


Salary and benefits questionnaire"^ 




X 








Head Start verification^ 




X 









X Round that included the instrument. 



* The fall-first grade data collection consisted of a 30 percent subsample of the study schools enrolling 27 percent of the eligible children. See the 
User’s Manual for the ECLS-K First Grade Public-Use Data Files and Electronic Code Book (NCES 2002-135) for information about the 
purposes and methods of the fall-first grade data collection. 

^ In spring-first grade, there were two sets of teacher questionnaires — one for the teachers of children who had made the transition to the first 
grade or any higher elementary school grade, and the second for teachers of children who were repeating or attending the second year of 
kindergarten. 

^ In spring-first grade, there were two different school administrator questionnaires — one for school administrators in schools new to the study 
and one for school administrators in schools that participated in the base year data collection. 

^ The salary and benefits questionnaire collected information on the base salary, merit pay, and health benefit pay of teachers and principals. It 
was completed by the school or district business administrator or by a private school administrator or headmaster. 

^ The Head Start Verification Study verified parent and school reports of children’s Head Start participation by matching information on the 
name and location of the Head Start facilities the children were reported to have attended against a database of Head Start centers. For each 
match, the center was contacted to confirm that the child had attended the center in the year before kindergarten. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 kindergarten, first, and third grade data collections, school years 1998-99, 1999-2000, and 2001-02. 
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Exhibit 1-4. Direct child assessments, by round of data collection: School years 1998-99, 
1999-2000, and 2001-02 



Direct child assessments 


1998- 

school 


1999 

year 




1999-2000 
school year 




2001-02 
school year 


Fall- 

kindergarten 


Spring- 

kindergarten 


Fall- 
first grade 


Spring- 
first grade 


Spring- 
third grade 


Eanguage screener 
















(Oral Eanguage Development 
















Scale [OEDS])‘ 


X 




/ 


/ 




/ 




Reading (language and literacy) 


X 




X 


X 




X 


X 


Mathematical thinking 


X 




X 


X 




X 


X 


Socioemotional development 














X 


General knowledge 
















(science and social studies) 


X 




X 


X 




X 




Science 














X^ 


Psychomotor 


X 














Eleight and weight 


X 




X 


X 




X 


X 



X Round that included the instrument. 



/ The OLDS was administered to language minority students who were new to the study in the spring or did not pass the cut score in the English 
version during the previous OLDS administration. 

' The OLDS was given to children with a non-English language to determine if the children understood English well enough to receive the direct 
child assessments in English. For further information on the OLDS, please refer to the Early Childhood Longitudinal Study, Kindergarten Class 
of 1998-99 (ECLS-K), Base Year Public-Use Data Files and Electronic Code Book: User’s Manual (NCES 2001-029, U.S. Department of 
Education, National Center for Education Statistics, 2000) or the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), 
Restricted-Use Base Year Child File, Teacher File, and School File (NCES 2000-097, U.S. Department of Education, National Center for 
Education Statistics, 2001). The OLDS was not used in third grade because the vast majority of children passed it by spring-first grade. 

^ In spring-third grade, general knowledge was replaced with a science assessment. Children received a science assessment that measured their 
understanding of science concepts and scientific investigation skills. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 kindergarten, first, and third grade data collections, school years 1998-99, 1999-2000, and 2001-02. 



Additional information about the ECLS program can be found on the NCES web site: 
http://www.nces.ed.gov/ecls. 



1.4 Second Grade Bridge Study 

One of the critical goals of the ECES-K is to measure children’s growth in cognitive 
achievement through the early elementary school years. Due to budgetary constraints, data were not 
collected during the 2000-01 school year, when most of the sampled children were in second grade. The 
absence of second grade data presented a challenge for establishing longitudinal scales to link the first 
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grade to third grade scores. Very few children answered the most difficult items in the spring-first grade 
data collection correctly. Third grade field test results indicated that these same items would be too easy 
for the vast majority of third graders. The ability levels of first graders overlapped with those of third 
graders only in the tails of the distributions. For this reason, a bridge study was conducted to fill in the 
data points that lie between the preponderance of the first and third grade ability levels so that stable item 
parameter estimates could be calculated that would support the measurement of gain. Reading and 
mathematics assessment data collected from a relatively small sample of second graders served as a 
bridge to link the first and third grade rounds. This bridge sample did not consist of ECLS-K longitudinal 
sample members, nor was it nationally representative. For more information about the bridge study, see 
ECLS-K Psychometric Report for the Third Grade. 



1.5 Contents of Report 

This report provides detailed technical information about the development, design, and 
conduct of the third grade data collection. Chapter 2 provides an overview of process used to develop the 
computer-assisted (CAI) and hard-copy survey instruments. Approximately 89 percent of the children 
interviewed were in third grade during the 2001-02 school year, 9 percent were in second grade, and less 
than 1 percent were in fourth grade or higher. Chapter 3 describes the sample design and implementation. 
Chapter 4 describes the data collection methods, including information about the training of field staff and 
quality control procedures. Chapter 5 details the preparation and editing of the data as it is receipted from 
the field. Chapter 6 provides information on unit and item response rates. Chapter 7 discusses weighting 
and variance information. A separate nonresponse bias analysis report examining whether survey 
nonresponse is adversely affecting the quality of the data will be available in fall 2003. 

Because both this report and the ECLS-K Psychometric Report for the Third Grade focus on 
the third grade data collection, minimal information is provided about the base year or first grade data. 
Users who wish to learn more about these data collections should refer to the following documents: 
ECLS-K Psychometric Report for Kindergarten Through First Grade (NCES 2002-05), ECLS-K Base 
Year Public-Use Data Files and Electronic Code Book: User’s Manual (NCES 2001-029); ECLS-K 
Restricted-Use Base Year: Child File, Teacher File, and School File (NCES 2000-097); User’s Manual 
for the ECLS-K First Grade Public-Use Data Files and Electronic Code Book (NCES 2002-135); or 
User’s Manual for the ECLS-K First Grade Restricted-Use Data Files and Electronic Code Book (NCES 
2002-128), all of which are produced by the U.S. Department of Education, National Center for 
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Education Statistics. Additional information about the ECLS program can be found on the World Wide 
Web at http://www.nces.ed.gov/ecls. 
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2. DEVELOPMENT OF SURVEY INSTRUMENTS 



The ECLS-K third grade survey collected data on the achievement and home and school 
experiences of children attending kindergarten in 1998-99 to provide information on the children’s 
progress in the middle elementary grades. In the design phase of the previous ECLS-K kindergarten and 
first grade waves of data collection, policymakers, teachers, and researchers were consulted, and relevant 
literature was reviewed to ascertain the specific areas within each of the topical components for which 
national data were needed. Information gathered from these activities guided the formulation of research 
questions deemed most important for the ECLS-K to address. Extant surveys were reviewed to identify 
surveys that had been fielded to answer similar questions. 

The ECES-K data collection instruments are similar in content and form in all five waves of 
the study. The ECES-K employs two modes of data collection, computer-assisted and self-administered 
hard-copy instruments. This chapter describes the development of the computer-assisted and hard-copy 
instruments for the third grade data collection. The procedures for developing the child assessment 
battery and indirect rating forms are described in a separate psychometric report. More information on 
the assessment battery and indirect rating forms is found in Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 (ECLS-K) Psychometric Report for Kindergarten Through First Grade 
(NCES 2002-05; U.S. Department of Education, National Center for Education Statistics, 2002) and in 
the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) Psychometric Report 
for the Third Grade (NCES 2004-087, forthcoming). 

The development process for the spring-third and spring-fifth grade parent, teacher, and 
school administrator instruments involved several stages. The first stage involved a review of the first- 
grade and kindergarten questionnaires to identify age-inappropriate items and new areas for inclusion. 
This was followed by reviews of extant data sources and relevant literature. Next, advice from the ECES- 
K Technical Review Panel and other consultants was sought. Finally, a timing study was conducted to 
ensure that undue burden was not placed on parent respondents. These different phases of the 
development process are discussed below. 
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2.1 



Review of ECLS-K Instruments From Previous Rounds 



To provide data comparability across rounds, the majority of the third grade items were 
taken from previous ECLS-K questionnaires. However, some topics in the previous questionnaires were 
no longer age-appropriate. In addition, new topics important to address in third grade needed to be 
identified. Thus the first step in developing the third grade questionnaires was to review the 
questionnaires used in the prior waves of data collections, extant literature and surveys, and suggest topics 
to be excluded from the third grade questionnaires and to identify new ones to be added. 



2.2 Review of Extant Data and Literature 

Review of Extant Data. Reviews of several other surveys were conducted to identify items 
that could be used to measure constructs that were new to the parent interview, teacher questionnaires, 
and school administrator questionnaires at third grade. To find existing survey items for the third grade 
parent interview, the following surveys were reviewed: 

■ The Early Childhood Longitudinal Study, Birth Cohort (ECLS-B); 

■ The National Adult Literacy Survey (NALS); 

■ The 2003 Parent and family Involvement Survey of the National Household 
Education Surveys Program (PLI-NHES: 2003); 

■ The National Survey of America’s Lamilies (1997, 1999) (NSAL); 

■ The National Education Longitudinal Study of 1988 (NELS:88); 

■ Early Head Start (EHS); and 

■ High School and Beyond (HS&B). 

To identify items for the third grade teacher and school questionnaires, the following surveys 
were examined: 



■ National Assessment of Educational Progress (NAEP) 2000 Grade 4 Teacher 
Questionnaires: Reading Teacher, Math and Science Teacher, Reading/Writing/Civics 
Teacher; 
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■ Third International Mathematics and Science Study (TIMSS) Grades 3 and 4: Math 
Teacher Questionnaire, Science Teacher Questionnaire; 

■ Schools and Staffing Survey (SASS), Public School Questionnaire, Public School 
Teacher Questionnaire, Public School Principal Questionnaire, 1999-2000; 

■ The NCES Fast Response Survey System (FRSS): Public School Teachers’ Use of 
Computers and the Internet; 

■ The NCES Fast Response Survey System (FRSS): Teacher Professional Development 
and Training; 

■ The NCES Fast Response Survey System (FRSS): Principal/School Disciplinarian 
Survey on School Violence; 

■ The National Evaluation of the Federal Class Size Reduction Program: Survey of 
School Principals (Abt Associates); and 

■ Teaching, Feaming, and Computing: Teacher Survey. 

Review of Literature. Most of the ECFS-K spring-third grade parent questionnaire 
consisted of items repeated from previous rounds of the ECFS-K in order to have multiple time points for 
comparison. The recommendations for the spring-third grade parent, school, and teacher questionnaires 
were based on the literature review and development work prepared for the ECFS-K kindergarten and 
first grade instruments. In addition, recent child development and related journals were reviewed to 
identify important constructs and topics of research interest pertinent to children in the middle elementary 
school years. For the school and teacher questionnaires, materials from U.S. Department of Education 
Regional Education Faboratories and organizations with expertise in school reform were reviewed in 
order to identify important topics to cover. 



2.3 Review Panels 

Studies with the scope, complexity, and importance of the ECFS-K require input from a 
number of individuals and organizations to address the data needs of policymakers and of those 
performing policy studies and educational research. In addition, consultations with practitioners, content 
area experts, and researchers are necessary to ensure that instruments accurately reflected curricular 
standards and practices. The ECFS-K project staff established and sought guidance from the Technical 
Review Panel, the Content Review Panel, and individual consultants. 
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2.3.1 



Technical Review Panel 



The Technical Review Panel was assembled to provide review and comment on such matters 
as the technical design and implementation of the ECLS-K and policy and research topics that are 
appropriate for the ECLS-K third and fifth grade data collections. The membership of the Technical 
Review Panel represents a broad range of nonfederal and federal experts in elementary education, 
educational, and family research and policy issues. 

Technical Review Panel Members: 

K. Alexander, Johns Hopkins University 
R. Duran, University of California at Santa Barbara 
N. Karweit, Johns Hopkins University 
D. Morrison, Georgetown Public Policy Institute 
J. Stallings, Texas A&M University 
D. Stipek, Stanford University 

U.S. Department of Education: Institute of Education Sciences, Office of Special Education Programs, 
and Office of English Language Acquisition 

U.S. Department of Agriculture: Economic Research Service, and Office of Analysis, Nutrition, and 
Evaluation 

U.S. Department of Health and Human Services, Administration on Children, Youth and Families 

2.3.2 Content Review Panel 

In addition to the Technical Review Panel, the Content Review Panel was established to 
provide expert review of the validity of the content and consistency of the items in the assessment battery 
with instructional practice. The panel included subject matter experts in reading, mathematics, and 
science as well as school assessment and evaluation administrators. 

2.4 Discussions with Consultants/Experts 

In addition, the contractor (Westat) sought advice from five consultants/experts on topics 
identified for inclusion in the third grade data collection. Specifically, more information on computer-use 
questions and school climate was obtained. 
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2.5 



New Items and Constructs Identified for Third Grade 



2.5.1 Parent Interview 

Several new construct areas were added for third grade. These were as follows: 

■ Children’s stressful life events (FSQ.3 lOa-f); 

■ Reading and math resources in the home (HEQ.021a-d); 

■ Reading behavior of the parent (HEQ.030a-d); 

■ Identification of persons inside and outside the household who help with homework 
(HEQ.091-HEQ.098); 

■ Access to the Internet at home and child use of the Internet at home (HEQ.045, 
HEQ.046); 

■ Number of breakfasts eaten by child and parent per week and location where child 
eats breakfast (HEQ. 1 1 5-HEQ. 118); 

■ Child’s transportation to school (HEQ. 124-HEQ. 127); 

■ Non-residential parents’ payment of bills other than child support (NRQ.252- 
NRQ.253); 

■ Parent concerns about child emotional behavior and related diagnoses (CHQ.350- 
CHQ.375) 

■ Common childhood health conditions (CHQ.560-CHQ.585); and 

■ Parents’ work and/or school requirements for receipt of TANF and/or food stamps 
(WPQ.105-WPQ.1060S, WPQ.125-WPQ.130OS). 

Other areas were reintroduced in third grade from other years of ECES-K. These were as follows: 

■ Whether family moved to neighborhood so that child could go to a particular school 
(PIQ.006); 

■ Whether school was assigned or chosen (PIQ.007); 

■ Neighborhood problems (HEQ.410a-e); 

■ Marital satisfaction (CFQ. 100); 

■ Parenting warmth/aggravation (DWQ.OlOa-h); 

■ Days scheduled to see nonresidential parent (NRQ. 1 14); 

■ Physical activity level of child (CHQ.700-CHQ.705, CHQ.730); 

■ Frequency of child’s aerobic exercise (CHQ.710-CHQ.726); 

■ Parental depression (PPQ. 1 00-PPQ.2 1 0); and 

■ Food security (FDQ.130-FDQ.250). 
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2.6 



Timing Study 



As with any study instrument, questionnaire length and respondent burden were issues of 
concern. A timing study was conducted for the draft parent questionnaire. Five Westat staff members 
conducted nine interviews with respondents who had previously volunteered to participate in studies 
being conducted by Westat. No attempt was made to recruit respondents representative of either racial or 
economic groups as the objective was to obtain an estimate of the length of the questionnaire rather than 
to examine how individuals interpreted the questions. Westat did attempt to select people who would go 
through the various questionnaire paths (e.g., married couples, single parents, one household with a 
nonresident father). All of the respondents were parents of third grade children. All interviews were 
conducted over the telephone using a paper version of the questionnaire. Interviewers used stopwatches to 
time the individual sections and to get an overall time for the interview. The interviewers stopped the 
watches for extended interruptions, such as a respondent having to take care of the needs of a family 
member. In most cases, the respondents were asked to answer questions in sections that required 
knowledge of data collected from an earlier wave of the data collection as if they had provided the 
information in a previous round of the survey. In only two interviews were respondents asked to complete 
sections, as would be the case with a new respondent. 

The revised paper version of the questionnaire took an average of 59 minutes and 27 seconds 
to complete. Table 2-1 summarizes the overall and section timings for each interviewer and across 
interviewers. The initials denote the five interviewers. Two interviewers completed more than one 
interview (e.g., interviewer JW completed three interviews, JWl, JW2, and JW3). 
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Table 2-1. Interviewer timings for the revised ECLS-K third grade parent interview, by interviewer and 
by section : School year 2001-02 



Section 








Interviewer 








Section 


JWl 


AFl 


EPl 


JW2 


EP2 


EP3 


JW3 


NVl 


RRl 


Average 


Total 


0:56:54 


1:06:53 


0:58:40 


0:48:40 


0:54:04 


0:54:18 


0:52:02 


1:05:00 


1:18:33 


0:59:27 


INQ 


0:02:01 


0:01:39 


0:02:19 


0:01:39 


0:02:12 


0:02:25 


0:02:05 


0:01:36 


0:02:05 


0:02:00 


PIQ 


0:09:29 


0:08:21 


0:10:37 


0:07:12 


0:08:03 


0:09:21 


0:07:50 


0:09:34 


0:11:15 


0:09:05 


ESQ 


0:01:40 


0:02:27 


0:05:24 


0:01:28 


0:03:15 


0:03:41 


0:01:01 


0:02:22 


0:01:00 


0:02:29 


HEQ 


0:15:53 


0:14:25 


0:13:53 


0:11:10 


0:13:48 


0:13:02 


0:13:55 


0:16:55 


0:20:05 


0:14:47 


SSQ 


0:02:57 


0:03:28 


0:03:00 


0:02:47 


0:02:53 


0:02:37 


0:03:19 


0:03:13 


0:03:00 


0:03:02 


CFQ 


0:01:01 


0:01:10 


0:01:01 


0:01:10 


0:01:18 


0:01:37 


0:02:02 


0:01:36 


0:02:00 


0:01:26 


CCQ 


0:04:01 


0:06:14 


0:01:04 


0:04:10 


0:01:30 


0:01:24 


0:01:09 


0:01:34 


0:02:00 


0:02:34 


DWQ 


0:05:09 


0:05:02 


0:03:27 


0:05:09 


0:04:23 


0:04:08 


0:03:53 


0:05:38 


0:07:07 


0:04:53 


NRQ 


0:00:00 


0:05:23 


0:00:00 


0:00:00 


0:00:00 


0:00:00 


0:00:00 


0:00:00 


0:00:00 


0:00:36 


CHQ 


0:07:35 


0:05:26 


0:09:05 


0:06:26 


0:07:48 


0:07:46 


0:09:00 


0:08:39 


0:11:37 


0:08:09 


PPQ 


0:01:49 


0:01:56 


0:01:49 


0:01:48 


0:01:51 


0:01:53 


0:02:33 


0:02:20 


0:03:29 


0:02:10 


FDQ 


0:02:17 


0:03:31 


0:01:49 


0:01:44 


0:01:35 


0:01:29 


0:01:38 


0:02:14 


0:02:15 


0:02:04 


PEQ 


0:00:39 


0:00:25 


0:00:27 


0:00:20 


0:00:23 


0:00:25 


0:00:39 


0:01:20 


0:02:00 


0:00:44 


EMQ 


0:00:28 


0:00:11 


0:00:15 


0:00:40 


0:00:23 


0:00:15 


0:00:25 


0:02:05 


0:03:00 


0:00:51 


WPQ 


0:00:39 


0:01:35 


0:01:25 


0:01:23 


0:01:05 


0:01:09 


0:00:55 


0:00:54 


0:03:42 


0:01:25 


PAQ 


0:00:38 


0:01:20 


0:00:41 


0:01:08 


0:01:04 


0:01:13 


0:00:50 


0:02:00 


0:02:00 


0:01:13 


CMQ 


0:00:38 


0:04:20 


0:02:24 


0:00:26 


0:02:33 


0:01:53 


0:00:48 


0:03:00 


0:01:58 


0:02:00 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten class of 1998-99 third grade data collection, school year 2001-02 timing study. 



Overall, the parent interview required just under an hour to complete. The timings ranged 
from a low of 48 minutes to a high of 78 minutes. The approaches used to capture the information (update 
versus obtain new data) and the characteristics of the child and household contributed to the variations in 
the length of interviews. It was estimated that the time to complete an interview for twins would extend 
the length of the interview, on average, by a conservative 30 minutes. This was determined by doubling 
the time for the three sections that required the most replication of data for each child (PIQ, HEQ and 
CHQ). 
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No individual section was unduly long. The section that required the most time to administer 
was the Home Environment, Activities and Cognitive Stimulation (HEQ) section. It took between 1 1 
minutes and 20 minutes, depending upon the interviewer. 

In most of the timing interviews the questions on education and occupation were updated (as 
if the information had been obtained in an earlier interview), which would be the more common approach 
in the national study; however, in two interviews the sections were asked as if the information was not 
previously known. The timing differences with those two strategies were as follows: updating the 
education and occupation questions took an average of 25 seconds while collection of new information 
required an average of 2 minutes and 6 seconds. Only one respondent completed the Nonresident Parent 
section of the questionnaire, which includes items about parents who do not live with their child, such as 
a question about the frequency of their contact with the child. Also, only one of the sample respondents 
included a parent of twins. The timing, however, was only done for the first interview because this 
represented the most common approach. The results of the timing study suggested that the parent 
questionnaire could be administered within 45 minutes. 
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3. SAMPLE DESIGN AND IMPLEMENTATION 



The Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) 
employed a multistage probability sample design to select a nationally representative sample of children 
attending kindergarten in 1998-99. In the base year the primary sampling units (PSUs) were geographic 
areas consisting of counties or groups of counties. The second-stage units were schools within sampled 
PSUs. The third and final stage units were students within schools. During the base year data were 
collected in both the fall and the spring. 

The first grade data collection included only base year respondents. A case was considered 
responding for the base year if there was a completed child assessment or parent interview in fall- or 
spring-kindergarten. A child with a disability who could not be assessed was also considered a base year 
respondent whether or not this child had a complete parent interview. Background characteristics such as 
gender, race/ethnicity, age, height, and weight are available for children with disabilities who could not be 
assessed. While all base year respondents were eligible for the spring- first grade data collection, the effort 
for fall-first grade was limited to a 30 percent subsample. The spring-first grade student sample was 
freshened to include current first graders who had not been enrolled in kindergarten in 1998-99 and, 
therefore, had no chance of being included in the ECES-K base year kindergarten sample. For both fall- 
and spring-first grade, a 50 percent subsample of students who had transferred from their kindergarten 
schools was followed. 

The third grade data collection included base year respondents and children sampled in first 
grade through the freshening operation. As in the first grade data collection, only a subsample of students 
who had transferred from their kindergarten schools was followed. In third grade, however, the 
subsampling rate applied to movers was slightly higher: children whose home language was non-English 
(also known as children belonging to the language minority group) and who moved for the first time in 
third grade were followed with certainty. In other words, 100 percent of the children belonging to the 
language minority group who did not move between kindergarten and first grade but moved between first 
grade and third grade were followed into their new third grade schools. Eanguage minority children who 
had moved between kindergarten and first grade and were not subsampled for followup in first grade did 
not reenter the third grade sample; those who were subsampled for followup in first grade were followed 
with certainty into their third grade schools if they moved again between first grade and third grade. The 
higher subsampling rate allowed for the preservation of this group in the sample for analytic reasons. 
Children not in the language minority group continued to be subsampled for followup at a 50 percent rate 
if they moved out of the original sample schools. 



3-1 




The precision requirements and achieved sample sizes for the different waves of data 
collection are discussed in section 3.1. The base year, fall- first grade, and spring-first grade samples are 
discussed in section 3.2, 3.3 and 3.4, respectively. Sampling issues that were considered prior to the third 
grade data collection are discussed in section 3.5. Section 3.6 discusses the characteristics of the third 
grade sample with an emphasis on children in the language minority group and children who moved out 
of their original sample schools since these two groups of children were the focus of the sampling issues 
discussed in section 3.5. 



3.1 Precision Requirements and Achieved Sample Sizes 

The ECLS-K is a nationally representative longitudinal survey of children who attended 
kindergarten in 1998-1999, supplemented with children who were in first grade in spring 2000, but were 
not in kindergarten the previous year. Data on these children were collected from a variety of sources at 
two points in the base year (kindergarten in 1998-1999), two points in the 1999-2000 school year (as 
noted earlier, the fall collection was limited to a subsample of children) when most of the children were in 
first grade, and again in spring of 2002 when most of the children were in third grade. The plan calls for 
collecting data on children again in spring of 2004 when most of the children are in fifth grade. 

The overall design for the survey evolved over time. The initial design study recommended 
sampling 23,500 children in approximately 1,000 kindergarten programs sampled from 100 PSUs. The 
initial plans also called for sampling children in private schools at a higher rate than children in public 
schools, as well as sampling minorities (children of Black, Hispanic, or Asian or Pacific Island [API] race 
or ethnicity) at higher rates than nonminorities. The design study assumed that because of nonresponse 
and losses due to children moving, the final number of completed interviews at the end of the survey 
would be about 10,300. While the design study was useful in providing overall direction, the final 
framework for the sample design differed in many ways from its recommendations. 

The sample design implemented through the third grade in the ECLS-K is described in this 
chapter. The remainder of this section gives an overview of the sampling objectives and how the design 
was revised to accommodate changes in those objectives over the course of the study. Subsequent 
sections of the chapter give the details of the procedures used to implement the sample in the various 
rounds or waves of data collection, beginning with the base year in 1998-1999. 
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Four precision requirements for the survey were identified and formed the basis for the base 
year sample design and plans for the followups in subsequent rounds. These requirements are the ability 
to do the following: 



■ Measure a change of 20 percent in proportions across waves; 

■ Measure a change of 5 percent in a mean score across waves; 

■ Estimate a proportion for each wave with a coefficient of variation (CV) of 1 0 percent 
or less; and 

■ Estimate a mean score for each wave with a CV of 2.5 percent or less. 

The goals were interpreted as being objectives not only for all children, but for subgroups of 
analytic interest including children attending public and private schools and children from different race 
and ethnic groups. After the spring-first grade data collection, language minority (EM) children were a 
newly identified subgroup of analytic interest for sample design purposes. A large number of assumptions 
had to be made to estimate sample sizes sufficient to meet the precision requirements. The key 
assumptions included projections of the losses due to nonresponse and attrition due to children moving, 
the design effects' associated with the sample design, the element mean and standard deviations of the 
scores, and the correlation of the statistics across waves. Since the ECES-K is the first study of this 
population using this methodology, many of the assumptions had to be based on judgments without much 
empirical data to support them. 

The precision requirements that drive the sample design (those demanding the largest sample 
size) have to deal with estimating changes over time and estimating the precision of estimates in the fifth 
grade data collection. Based on assumptions described above, it was determined that a sample in fifth 
grade of about 10,000 children would be adequate to meet the precision requirements overall and for most 
subgroups. A sample of about 800 to 1,000 children in a subgroup would be achieved for most of the 
subgroups with an overall sample of 10,000 children and these would approximately meet the precision 
goals. Children in private schools and APIs were the two subgroups that were expected to fall short of the 
goals if higher sampling rates were not applied. As noted in the following sections, sampling procedures 
were implemented to increase the sample size for these two groups. 

After the spring-first grade data collection was completed, the assumptions were reviewed 
and the ability of the sample to meet the survey goals was re-examined. At that time, language minority 



' When a clustered sample with unequal sampling weights is used, the estimates are less precise than those expected from a simple random 
sample and the ratio of the actual to simple random sampling variance is called a design effect. 
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children were included as being a subgroup of analytic interest. The evaluation showed that the sample 
sizes were adequate for most subgroups, but special efforts were needed to retain API children and 
language minority children in subsequent rounds. Table 3-11 in section 3.5 shows the outcome of the 
spring-first grade data collection by type of children. Since funding was made available to support these 
efforts, sampling procedures for retaining movers were modified. In the first grade data collection, half of 
the movers were subsampled and included for followup, without taking any characteristics of the children 
into account. To increase the sample of API and language minority children, the sampling procedures 
were revised for the third grade followup to retain as many of these children as possible. 

The evaluation also showed that the assumed design effects for test scores (reading, math 
and general knowledge) were larger than originally expected, ranging from 4.5 to 9.5. The larger than 
expected design effects for scores were first identified after the base year. The design effects for 
percentages, ranging from 1.6 to 6.9 for proportions greater than 30 percent, were close to those originally 
anticipated (3.8 on average).^ The design effects for scores affect the ability to meet the two precision 
goals for scores. The evaluation also showed the correlation over time of the scores was higher than 
expected, having the opposite effect and making estimates of change in scores over time more precise. 
Thus, only one precision objective is adversely affected by the higher than expected design effects for 
scores. It may mean that estimates of mean score for the last wave of data collection (i.e., fifth grade) for 
smaller subgroups will have CVs of more than 2.5 percent. 

Table 3-1 tracks the ECLS-K sample from the base year through third grade. The table 
shows that the large initial sample of children has been reduced over time due to subsampling movers and 
nonresponse. While the initial assumptions that drove the sample design were not always accurate 
separately, the overall effect of the losses has been very close to what was expected. The overall number 
of eligible children at the end of the third grade wave was nearly 17,000 children compared to the 
expected value of 17,000 to 18,000 using assumed mover rates as discussed later in table 3-12. Projecting 
the losses forward (i.e., 45 percent of children who are still in the original school by the end of third grade 
will move to a new school during fifth grade and 90 percent of the eligible children will have completed 
assessment in fifth grade), it is likely that the final sample size for the fifth grade sample will exceed the 
10,000 children in the initial projections. In fact, depending on the response and mover rates associated 
with the transition between third and fifth grade, the overall sample size will be approximately 11,000 
children. 



^ See design effects for selected survey items in chapter 4 of the Early Childhood Longitudinal Study, Kindergarten Class of 1998—99 (ECLS-K), 
Base Year Public-Use Data Files and Electronic Code Book: User’s Manual (NCES 2001-029: U.S. Department of Education, National Center 
for Education Statistics, 2001). 
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Table 3-1, ECLS-K sample size from the base year through third grade, by selected 
characteristics: School years 1998-99, 1999-2000, and 2001-02 



Characteristic 


Fall- 

kindergarten 


Spring- 

kindergarten 


Fall- 
first grade* 


Spring- 
first grade 


Spring- 
Third grade 


Beginning wave sample size 


21,387 


22,772^ 


6,507 


21,357 ^■"* 


21,357 


Fielded after subsampling movers 






5,728 


18,507 


17,240 


Fielded after locating movers 




22,088 


5,691 


17,708 


16,951 


Number of eligibles 


21,356 


21,941 


5,652 


17,652 


16,829 


Child-complete^ 


19,173 


19,967 


5,291 


16,727 


14,470 


Parent-complete® 


18,097 


18,950 


5,071 


15,626 


13,489 


Child- or Parent-complete 


19,864 


20,578 


5,424 


17,324 


15,305 


Child- and Parent-complete 


17,586 


18,339 


4,938 


15,029 


12,654 



' Only 30 percent of base year schools were included in the fall-first grade sample. 

^ Including 1,426 children from refusal converted schools and excluding 41 children in schools that cooperated in fall-kindergarten 
and refused in spring-kindergarten. 

^ Only children who have at least one of the four base year data points (fall-kindergarten assessment or parent data, or spring- 
kindergarten assessment or parent data). 

Including 165 children sampled in first grade through sample freshening. 

^ Child-complete if the child had assessment data or was not assessed due to a disability. 

^Parent-complete if the child had parent data. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 kindergarten, first, and third grade data collections, school years 1998-99, 1999-2000, and 2001-02. 



The details on the sample sizes for subgroups at the end of the third grade are provided later 
in this chapter (see tables 3-14, 3-15, and 3-16). Those tabulations show that the third grade sample sizes 
for all of the identified subgroups of interest exceed 1,100. For many of the groups the sample sizes are 
much larger, with the lowest sample size for API children. Thus, the sampling and data collection 
procedures developed in the initial stages and modified throughout the course of the study are producing 
samples that are meeting or exceeding the requirements. The final outcomes in fifth grade depend on the 
success of the operations and procedures proposed for that year, but these results suggest the prospects of 
achieving the sample size goals are excellent. 



3.2 Base Year Sample 

In the base year, the ECLS-K selected a nationally representative sample of children 
attending kindergarten in 1998-99 using a dual-frame multistage probability sample design. Counties and 
groups of counties constituted the first stage sampling units or PSUs, schools or kindergarten programs 
within PSUs were the second-stage units, and children were the third- and final-stage units. 
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3.2,1 



Selecting the Area Sample 



The point of departure for the ECLS-K area sample frame development was an existing 
multipurpose frame of PSUs created using 1990 county-level population data and 1988 per capita income 
data from the U.S. Department of Commerce, Bureau of Economic Analysis. This frame contained 1,404 
PSUs that were counties or groups of contiguous counties. PSUs did not cut across census regional 
boundaries, but were allowed to cross state boundaries. Each 1990 metropolitan statistical area (MSA)"^ 
constituted a single PSU except where an MSA crossed census regions, and it was split into two PSUs. 
The minimum size of a PSU in the multipurpose frame was 15,000 persons. 

Since the focus of the ECES-K is kindergarten students, the existing PSU frame was updated 
with 1994 population estimates of 5-year-olds by race/ethnicity, the most up-to-date estimates available 
from the U.S. Bureau of the Census at the time. The counts of 5-year-olds by race/ethnicity were used to 
revise PSU definitions relative to a different minimum PSU size and to construct a measure of size 
(MOS) that facilitated the oversampling of APIs. Each PSU in the frame that did not have at least 320 5- 
year-olds was collapsed with an adjacent PSU. This minimum PSU size was developed based on 
assumptions concerning anticipated school response rates, the average number of schools that would be 
selected per PSU, and the target number of students to be sampled per school. After this collapsing, the 
final ECES-K PSU frame contained 1,335 records. 

The MOS used for selecting PSUs took into account the amount of oversampling of APIs 
required to meet the ECES-K precision goals. The weighted MOS was calculated as follows: 

MOS = 2.5 X 

where 2.5 is the oversampling rate for APIs, and and are the counts of 5-year-old APIs and all 
others, respectively. The oversampling rate for APIs was calculated as the target number of completed 
API cases divided by the expected number of completed API cases without oversampling. In all, 100 
PSUs were selected for the ECES-K. The 24 PSUs with the largest measures of size were designated as 
certainty selections or self-representing (SR)^ and were set aside. They were included in the sample with 
certainty. Once the self-representing PSUs were removed, the remaining PSUs, called non-self- 
representing (non-SR)^, were partitioned into 38 strata of roughly equal MOS. The frame of non-self- 
representing PSUs was first sorted into eight superstrata by MSA status and by census region. Within the 



^ A census region is a geographic region defined by the U.S. Bureau of the Census. 

A metropolitan statistical area (MSA) is a geographic entity designated as one or more counties in a metropolitan area, except in New England, 
where MSA is defined in terms of county subdivisions. MSAs generally have under 1 million in population. 

^ A self-representing PSU is selected into the sample with certainty (i.e., with probability 1). 

^ A non-self-representing PSU is selected into the sample with probability proportional to its measure of size (MOS). 
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four MSA superstrata, the variables used for further stratification were race/ethnicity (high concentration 
of API, Black, or Hispanic), size class (MOS > 13,000 and MOS < 13,000), and 1988 per capita income. 
Within the four non-MSA superstrata, the stratification variables were race/ethnicity and per capita 
income. Table 3-2 describes how the 38 non-self-representing strata were created. 

Table 3-2. Stratum definitions for the 38 non-self-representing strata: School year 1998-99 



Stratum 



Metropolitan 
statistical area 
(MSA) status 



Census 

region* 



PSU measure £ 

Race/ethnicity in percent of size (MOS) 



Per capita income range 





' Non-MSA 


Northeast 


Any 


Any 


Any 


A 


; Non-MSA 


Midwest 


Any 


Any 


14,124 


17,4 


• Non-MSA 


Midwest 


Any 


Any 


13,277 


14,1 


1 Non-MSA 


Midwest 


Any 


Any 


12,169 


13,2 


Non-MSA 


Midwest 


Any 


Any 


6,992 


12,1 


: Non-MSA 


South 


Black > 42 


Any 


Any 


A 


; Non-MSA 


South 


25 < Black < 42 


Any 


Any 


A 


i Non-MSA 


South 


Any 


Any 


12,727 


20,0 


i Non-MSA 


South 


Black < 25 


Any 


11,165 


12,6 


1 Non-MSA 


South 


Any 


Any 


6,018 


11,1 


' Non-MSA 


West 


Any 


Any 


12,887 


23,2 


; Non-MSA 


West 


Any 


Any 


6,959 


12,8 




A census region is a geographic region defined by the U.S. Bureau of the Census. Primary sampling unit. 

NOTE: In this table, “Any” means any value of the column variable. For example, stratum 1 includes PSUs that have MSA status, are located in the Northeast region, 
with a MOS greater than or equal to 13,000 and per capita income ranging between $22,062 and $25,424, and can have any value of the race/ethnicity percentage. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten 
data collection, school year 1998-99. 
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Two PSUs were selected from each non-self-representing stratum using Durbin’s Method 
(Durbin, 1967). This method selects two first-stage units per stratum without replacement, with 
probability proportional to size and a known joint probability of inclusion. The Durbin method was used 
because it has statistical properties that make it easier to compute variances. Table 3-3 summarizes the 
characteristics of the ECLS-K PSU sample. 

Table 3-3. Distribution of the ECLS-K primary sampling unit (PSU) sample by self-representing status, 
metropolitan statistical area (MSA) status, and census region: School year 1998-99 



Self- 

representing 
(SR) status 


Metropolitan 
statistical area 
(MSA) status 






Census region' 




Total 


Northeast 


Midwest 


South 


West 


Total 




100 


18 


25 


34 


23 


SR 


MSA 


24 


6 


5 


6 


7 


Non-SR 


MSA 


52 


10 


12 


18 


12 


Non-SR 


Non-MSA 


24 


2 


8 


10 


4 



' A census region is a geographic regioin defined by the U.S. Bureau of the Census. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 kindergarten data collection, school year 1998-99. 



The Durbin method required two passes of the frame with a different selection probability at 
each pass to obtain the desired probabilities of inclusion and joint probabilities of inclusion. In the first 
pass, one PSU was selected in the stratum with probability p^. In the second pass, the selected PSU was 

excluded and another PSU was selected with probability proportional to 

' 1 1 " 

Pi + 

where pi = M^/M and P 2 = M 2 IM, is the MOS of the first unit selected, M 2 the MOS of the second 
unit selected, and M the MOS of the stratum. 



The overall selection probability of non-self-representing unit i is 



Pi 



2M. 



i = \,2. 



The joint probability of inclusion of the first and second units is 



2PiPi 



1 1 



l-2/>j l-2p 



2 J 



k=i i 



Pk 



^i^-2p 






k ) 
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3.2.2 



Selecting the School Sample 



In the second stage of sampling, public and private schools offering kindergarten programs 
were selected. For each ECLS-K PSU, a frame of public and private schools offering kindergarten 
programs was constructed using existing school universe files: the 1995-96 Common Core of Data (CCD) 
and the 1995-96 Private School Universe Survey (PSS). The school frame was freshened in the spring of 
1998 to include newly opened schools that were not included in the CCD and PSS and schools that were 
in the CCD and PSS, but did not offer kindergarten according to those sources. A school sample 
supplement was selected from the supplemental frame. 



3.2.2. 1 School Frame Construction 

The 1995-96 CCD Public School Universe File was the primary source for the ECLS-K 
public school sampling frame. Most schools run by the U.S. Department of the Interior, Bureau of Indian 
Affairs (BIA) and the schools run by the U.S. Department of Defense (DOD) are not included on the 
CCD. The 1995-96 Office of Indian Education Programs Education Directory was consulted in order to 
complete the list of BIA schools in the CCD file. For the DOD schools, a 1996 list of schools obtained 
directly from the DOD was used. The 1995-96 PSS Universe File was used as the primary source of the 
private school sampling frame. 

The first step in frame construction involved subsetting the file to schools located in counties 
that constituted the ECLS-K PSU sample. Further subsetting retained only those schools that offered 
transitional kindergarten, kindergarten, or transitional first grade, or which were strictly ungraded, as 
indicated by the school’s grade span. 

The constructed ECLS-K school frame included 18,911 public-school records and 12,412 
private-school records. The school frame was supplemented in the spring of 1998 to include schools that 
would be operational in fall 1998 but that were not included in the frame just described. The procedures 
used to supplement the frame are given later in this section. 

Table 3-4 gives the estimated the number of schools offering kindergarten programs and the 
number of kindergarten students from the ECLS-K school frame. These are the numbers of schools and 
students in the sampled PSUs in the frame weighted by the inverse of the PSU selection probabilities. 
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Table 3-4. Estimates of kindergarten schools and students, by primary sampling unit (PSU) status: 
School year 1998-99 



Number of kindergarten schools Number of kindergarten students 





Total 


Public 


Private 


Total 


Public 


Private 


Total 


73,095 




23,011 


4,089,781 


3,521,040 


568,741 


Self-representing PSUs 


19,721 


11,283 


8,438 


1,277,419 


1,059,535 


217,884 


Non-self-representing PSUs 


53,374 




14,573 


2,812,362 


2,461,505 


350,857 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 kindergarten data collection, school year 1998-99. 



3.2.2.1 School Measure of Size 

Within each PSU, schools with fewer than a predetermined minimum number of 
kindergarten students were clustered together before sampling in order to obtain a sample that is closer to 
self-weighting. The minimum number of kindergartners was 24 for public schools and 12 for private 
schools. Schools were selected with probability proportional to size. As with the PSU sample, a weighted 
MOS was constructed taking into account the oversampling of APIs: 

SCHMOS.J =2.5x n^„ .j 

where 2.5 is the oversampling rate for APIs, and njpjjj and are the counts of API kindergarten 

students and all other kindergarten students, respectively, in school j of PSU i. 



3.2.2.3 School Allocation 

Schools were sampled at rates designed to result in an approximately self-weighting sample 
of students within public and private school strata. The target number of sampled schools per PSU was 
calculated separately for public schools and private schools, and for self-representing and non-self- 
representing PSUs. The number of schools selected was the target number of schools adjusted upward by 
the estimated school response and eligibility rate. 
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3.2.2,3,1 Public Schools 



The total MOS for public schools was partitioned into the self-representing and non-self- 
representing strata. There are 1 00 PSUs in the ECLS-K sample, of which 24 are in the self-representing 
strata. The number of public schools selected from the self-representing strata was calculated as 

24 

Y,w,xPSUMOS^ 

nsR =Ti5 

^w.xPSUMOS^ 

1=1 

where n is the total number of public schools to be selected, Wj is the weight of PSU i, and 

PSUMOS, = y SCHMOS, . 

j 

The value for n is 800/. 85 = 941 where .85 is the expected eligibility and response rate for 
public schools. The supplement of public schools was expected to add relatively few schools to the frame 
and thus the 85 percent rate was not modified. The distribution of sampled schools was approximately 
291 for self-representing strata and 650 for non-self-representing strata. For self-representing and non- 
self-representing strata alike, the number of schools allocated to each PSU was proportional to the 
weighted MOS of the PSU {WixPSUMOSj). 

In the ECLS-K public school frame, 4 percent of public schools had fewer than 24 
kindergarten students. These schools were combined with other schools in the same PSU to form clusters 
with at least 24 students prior to sampling. Schools with 24 students or more were not grouped, but were 
also referred to as clusters (of one school each). To sample approximately 941 public schools, around 915 
clusters (single schools or groups of schools) have to be selected. As a general rule, if a sampled school or 
cluster of schools had 24 or more students, 24 students were selected. However, for practical reasons, all 
students in the sampled school or cluster were selected if there were fewer than 27 students. More details 
on the clustering of schools are found in the next section. 

The number of clusters was allocated to each PSU proportionally to the weighted MOS of 
the PSU (wixPSUMOSj). When the 915 clusters were allocated to PSUs, it was discovered that in 5 PSUs 
there were not enough clusters in the frame to select the required number of clusters. This resulted in only 
900 clusters being selected. Table 3-5 shows the expected distributions of clusters, schools, and students. 
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Table 3-5. Expected number of clusters, schools, and students — public schools: School year 1998-99 



Type of primary 
sampling unit (PSU) 


Number 
of clusters 
to select 


Expected 
number 
of schools 
sampled 


Expected 
number 
of students 
sampled 


Average 
number of 
students/ 
school 


Total 


900 


944 


21,643 


23 


Self-representing PSUs 


283 


285 


6,792 


24 


Non-self-representing PSUs 


617 


659 


14,851 


23 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 kindergarten data collection, school year 1998-99. 



3.2.2.3.2 Private Schools 

The same procedure was used to determine the allocation of the private schools. The private 
school target samples are labeled n^^'and njV 57 ?'for self-representing and non-self-representing PSUs 
respectively, and n' is the sum of ngu! and njgSR'- The value of n' is 200/.60=333, where .60 is the 
expected eligibility and response rate. The supplement to the frame was expected to add some private 
schools with kindergarten programs. The rate 60 percent was used because of the uncertainties associated 
with the estimate of the eligibility and response rate for private schools. 

The percentage of schools with fewer than 24 kindergarten students was large for private 
schools. Approximately 56 percent of private schools offered a kindergarten program that had fewer than 
24 students. Schools having fewer than 12 kindergarten students (according to the frame) were grouped 
into clusters of schools with at least 12 students in each cluster, following the clustering rules discussed in 
the next section. Schools with 12 students or more were not grouped. As a general rule, if a sampled 
school or cluster of schools had 24 or more students, 24 students were selected; if a sampled school or 
cluster had fewer than 24, all students were sampled. However, for practical reasons, all students in the 
sampled school or cluster were selected if there were fewer than 27 students. 

In order to sample approximately 333 private schools, 278 clusters were selected (single 
schools or groups of schools). Table 3-6 shows the expected distributions of clusters, schools, and 
students. 
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The number of clusters was not allocated separately to each self-representing PSU, since 
sampling was done on the aggregated list of schools in the self-representing PSUs. For the non-self- 
representing PSUs, the sample was allocated to each PSU proportionally to the weighted MOS of the PSU 
(WjxPSUMOS,), with a minimum of one cluster per PSU imposed if the PSU was so small that it was not 

allocated any clusters. 



Table 3-6. Expected number of clusters, schools, and students — private schools: School year 1998-99 



Type of primary 
sampling unit (PSU) 


Number 
of clusters 
to select 


Expected 
number 
of schools 
sampled 


Expected 
number 
of students 
sampled 


Average 
number of 
students/ 
school 


Total 


278 


333 


6,336 


19 


Self-representing PSUs 


107 


125 


2,456 


20 


Non-self-representing PSUs 


171 


208 


3,880 


19 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 kindergarten data collection, school year 1998-99. 



3.2.2A Clustering of Small Schools 

As noted above, schools with fewer than 24 students (public) or 12 students (private) were 
clustered together in order to obtain a sample that was closer to self-weighting. For example, if a school 
with 12 students was not clustered, the students from that school would be sampled at about half the 
probability as students in larger schools. The goal of the clustering of small schools was to form school 
clusters with a small number of schools that have close to 24 students and are heterogeneous. This goal 
was set so that if a cluster was selected, it would not be necessary to recruit many small schools; 
furthermore, the heterogeneity of schools improves the reliability of the estimates. Fleterogeneity was 
defined by school size for public schools, and by religious affiliation and school size for private schools. 
Within each PSU, schools with fewer than a predetermined minimum number of kindergarten students 
were separated from the frame and clustered together. A few exceptions to this general rule did occur and 
are discussed later. The procedures for clustering of schools are described below. 



3.2.2,4,1 Public Schools 

Schools with fewer than 24 kindergarten students were clustered. Within each PSU, the list 
of small schools was sorted in ascending order of kindergarten enrollment; it was then split in half, with 
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the second half re-sorted in descending order. The two halves were then put together in an interleaving 
fashion. Beginning at the top of the list, clusters of schools with at least 24 kindergarten students were 
formed. If the last cluster on the list still did not have the required 24 minimum, then it was put together 
with the next to the last cluster on the list. 



This clustering scheme resulted in 18 clusters with 5 or more schools, which were 
considered problematic as far as fieldwork was concerned. The worst case was one cluster with 13 
schools and only 41 students. In order to minimize the number of clusters having 5 or more schools, each 
problematic cluster was broken into groups of 2 or 3 schools, and each group was combined with the 
smallest of the “large” schools having 25 or more kindergarten students. Since enrollment in schools with 
missing kindergarten enrollment was imputed to be equal to 24, grouping any of these imputed schools 
with another school was avoided, lest they might turn out not to have kindergarten students. 



In addition to the 18 problematic clusters above, there were 12 PSUs with only 1 small 
school (with fewer than 24 kindergarten students) and there were 2 PSUs with only 2 small schools that, 
when grouped together, still had fewer than 24 kindergarten students. These small schools or groups of 
small schools were manually combined with the smallest school in the PSU having 25 or more students 
(see table 3-7). 

Table 3-7. Number of clusters and schools in the public school 
frame: School year 1998-99 



Number of schools 
in cluster 


Number of clusters 


Number of 
schools 


Total 


18,399 


18,911 


1 


18,095 


18,095 


2 


153 


306 


3 


97 


291 


4 


51 


204 


5 


3 


15 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early 
Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten data collection, 
school year 1998-99. 



3.2.2.4.2 Private Schools 



Schools with fewer than 12 kindergarten students were clustered. Within each PSU, the list 
of private schools was first sorted by religious and nonreligious affiliation. If the number of religious 
schools and nonreligious schools in the PSU differed by no more than a factor of 3, the smaller of the two 
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lists (religious or nonreligious) was sorted in descending order while the larger of the two lists was sorted 
in ascending order of kindergarten enrollment. The two lists were then put together in an interleaving 
fashion, so that the records that were at the bottom of the longer list were records with larger kindergarten 
enrollment, and did not have to be grouped together. Beginning at the top of the entire list, clusters of 
schools of at least 12 kindergarten students were formed. If the last cluster on the list still did not have the 
required minimum size, it was put together with the next to last cluster on the list. 

If the number of religious schools and nonreligious schools in the PSU differed by a factor 
greater than 3, schools were not separated into religious and nonreligious lists. Instead, the entire list of 
schools was sorted in ascending order of kindergarten enrollment; it was then split in half, with the second 
half re-sorted in descending order. The two halves were then put together in an interleaving fashion. 
Clusters of schools were formed as above. 

There were 3 PSUs where the clustering of small schools as specified above did not work 
well. Two of the 3 PSUs had only 1 small school each and the third one had 2 small schools that, when 
grouped together, still had fewer than 12 kindergarten students. These small schools or groups of small 
schools were manually combined with other large schools in the PSU (table 3-18). 



Table 3-8. Number of clusters and schools in the private school 
frame: School year 1998-99 



Number of schools 
in cluster 


Number of 
clusters 


Number of schools 


Total 


9,955 


12,412 


1 


7,640 


7,640 


2 


2,184 


4,368 


3 


121 


363 


4 


9 


36 


5 


1 


5 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early 
Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten data collection, 
school year 1998-99. 



3.2.2. 5 Implicit Stratification of Schools/Clusters of Schools 

Public schools with more than 24 kindergarten students and private schools with more than 
12 kindergarten students were not clustered. However, they are referred to as clusters (of one school each) 
for simplicity. 
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3.2.2. 5.1 Public Schools 



Within each PSU, the clusters were sorted by the MOS and separated into three size classes 
of roughly equal size (high, medium, and low). Within each size class, clusters were sorted by the 
proportion of APIs in a serpentine manner (alternating the sort order from one size class to the next). 



3.2.2, 5.2 Private Schools 

Within each PSU, each cluster was identified as religious, mixed, or nonreligious.’ The list 
of clusters was then sorted by these three categories. Within each category, the clusters were sorted in a 
serpentine manner by the MOS prior to selection. However, for the self-representing PSUs, all clusters 
were sorted as if they were from the same PSU, i.e., the aggregated list of clusters from the 24 self- 
representing PSUs was sorted by religious affiliation (religious/mixed/nonreligious). This procedure 
provided better control of the sample distribution of religious/mixed/nonreligious clusters. Across non- 
self-representing PSUs, clusters were sorted by religious affiliation, and within each category of religious 
affiliation, by the MOS in a serpentine manner. 



3.2.2. 6 School Selection 

Selection of the clusters of schools was systematic, with probability proportional to the 
MOS. Sampling of public schools was done independently within PSU (i.e., each PSU forms a separate 
sampling stratum) after the clusters of schools were sorted by MOS and proportion of API. Sampling of 
private schools was done separately for self-representing PSUs and for non-self-representing PSUs. All 
self-representing PSUs were placed in one sampling stratum and all non-self-representing PSUs were 
placed in a second stratum. In the self-representing stratum, sampling was done with one random start 
after sorting clusters of schools by religious affiliation and MOS. In the non-self-representing stratum, 
sampling was done with one random start after sorting clusters of schools by PSU, religious affiliation, 
and MOS. 



^ A cluster is “religious” if all schools in the cluster are Catholic schools or non-Catholic religious schools; “nonreligious” if all schools in the 
clusters have no religious affiliation; “mixed” if it has a combination of schools with or without religious affiliation. 
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3.2.2.7 



The ECLS-K Main School Sample 



A total of 1,280 schools were selected for the ECLS-K, of which 934 were public and 346 
were private schools. The characteristics of the school sample are presented in table 3-9. 



Table 3-9. Number of sample schools, by region and other characteristics: School year 1998-99 



Characteristic 


Total 


Public 


Private 


Total 


1,280 


934 


346 


Region 


Northeast 


238 


166 


72 


Midwest 


297 


215 


82 


South 


420 


309 


111 


West 


325 


244 


81 


Urbanicity 


Large central city 


245 


164 


81 


Mid-size central city 


252 


176 


76 


Urban fringe of large city 


386 


273 


113 


Urban fringe of mid-size city 


98 


78 


20 


Large town 


32 


25 


7 


Small town 


107 


80 


27 


Rural 


160 


138 


22 


Kindergarten enrollment 


<25 


210 


55 


155 


25-49 


224 


no 


114 


50-99 


467 


400 


67 


100-149 


236 


228 


8 


150-199 


88 


86 


2 


200 - 249 


26 


26 


0 


250-299 


15 


15 


0 


>300 


14 


14 


0 


Religious affdiation (private schools only) 


Catholic 


117 


t 


117 


Other religious 


143 


t 


143 


Nonreligious private 


86 


t 


86 


Tree lunch program 


Low (<=25% eligible students) 


284 


284 


t 


Medium low (>25% and <=50%) 


169 


169 


t 


Medium high (>50% and <=75%) 


122 


122 


t 


High (>75%) 


118 


118 


t 


Unknown 


241 


241 


t 



t Not applicable. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 kindergarten data collection, school year 1998-99. 
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3.2.2. 8 Supplemental School Sample 



As mentioned earlier, the public and private school frames were supplemented in the spring 
of 1998. The procedures for supplementing the frames were different for public schools, Catholic schools 
and non-Catholic private schools. These procedures are discussed below separately. 



3.2.2. 8.1 Public Schools 

Each public school district having one or more schools sampled was sent a sampling frame- 
based list of all schools offering kindergarten. Districts were asked whether any school expected to offer 
kindergarten in academic year 1998-1999 was missing from the list. For each school identified by the 
district, school name, address, telephone number, grade span, and kindergarten enrollment were obtained. 
Districts were also contacted that fell within the boundaries of the ECLS-K PSUs, but for which the CCD 
file listed no schools offering kindergarten, unless it was clear from their name that they were strictly 
secondary school districts (e.g., Middlebury Union Eligh School District). The information obtained from 
the school districts was checked against the ECES-K public school frame to confirm that these schools 
were truly new or newly eligible. Bona fide new schools were given a chance of being sampled. A new 
school’s chance of selection was conditioned on the school district’s probability of selection. Overall, 252 
new public schools were identified. Of these, 1 9 were selected using systematic sampling with probability 
proportional to size where the MOS was the same as it was for schools sampled from the main sample. 
Thus, a total of 953 public schools were included in the sample (934 -i- 19). 



3.2.2. 8.2 Private Schools 

The procedure for obtaining new school information from Catholic dioceses was exactly the 
same as for public schools. Since a diocese could cut across county or even state lines, each school 
identified by a diocese had to be associated with the correct county, and hence the correct PSU, before 
checking to see whether it was truly new. Since dioceses might cross PSU boundaries, a new Catholic 
school’s chance of being sampled had to be conditioned on the diocese’s probability of selection within 
the PSU where the new school was located. There were 126 new Catholic schools identified, and 6 were 
selected using systematic sampling with probability proportional to size. When combined with the main 
sample, the final Catholic school sample size was 123 (1 17 -l- 6). 
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3.2.2. 8.3 Non-Catholic Private Schools 



The search for non-Catholic private schools was considerably more complicated. Three 
classes of schools that had previously not been given a chance of selection from the PSS were 
reconsidered. Those were schools that had an unknown grade span because they had not responded to the 
1995-96 PSS, those that responded but did not report offering kindergarten, and those that appeared for 
the first time on the 1997-98 PSS file. Together these accounted for 2,544 potential new non-Catholic 
private schools. Beyond these additions from PSS, procedures similar to those used by the Bureau of the 
Census in the PSS area frame search were followed. These procedures included collecting lists of schools 
from different sources, matching them against the PSS list frame to remove duplicates, and further 
screening by telephone to verify new school status. The majority of new schools found by the Bureau of 
the Census for PSS came from telephone book yellow page listings. The yellow pages search was the 
main source of new non-Catholic private schools in the ECLS-K as well, yielding an additional 8,861 
possible new private schools. Since the number of kindergartners enrolled in these schools was unknown, 
a minimum kindergarten enrollment was assumed for sampling purposes (typically 24, unless the name 
was suggestive of day care in which case 12 was assumed). 

The supplemental frame contained 11,405 private schools. A sample of 279 schools was 
selected using systematic sampling with a probability proportional to these imputed enrollments. Each 
sampled school was contacted by telephone and screened to ascertain whether the school was public or 
private, whether it would be open in academic year 1998-1999; and if it would offer kindergarten. If the 
school met all of these conditions and was not Catholic, the school was eligible and released for data 
collection. 



A second supplemental procedure involved contacting local education agencies (EEAs) and 
local government offices for information on non-Catholic private schools. This procedure was done only 
in the smallest ECES-K PSUs, on the theory that if these PSUs had coverage problems their large weights 
were likely to introduce a larger bias in the estimates. All EEAs within these PSUs were contacted by 
telephone. For each city/town within the PSU, a list of local government offices was compiled using the 
blue pages. Successive government offices were called within a city or town until one was found that 
could provide information on private schools. As with the yellow pages, new schools identified by EEAs 
and local government offices were unduplicated against the PSS file before being added to the new school 
frame. Since kindergarten enrollment was unknown, it was imputed as described in the previous 
paragraph and sampling was performed using systematic sampling with probability proportional to size. 
The TEA search resulted in the identification of 30 new private schools after unduplication, of which 14 
were sampled. The local government search yielded 19 new schools, of which 8 were sampled. Finally, 
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three additional new private schools were reported by field staff based on personal knowledge. Of these, 
two schools were sampled. The same screening procedures to ascertain whether the school was public or 
private; whether it would be open in academic year 1998-1999; and if it would offer kindergarten were 
then applied to these sampled schools. 

The total number of non-Catholic private schools that were sampled was 303. After the 
screening procedures were applied, only 109 of these schools were eligible. These 109 schools are 
referred to as the supplemental sample of non-Catholic private schools. 

The final ECLS-K school sample for the base year was 1,414 schools, including 953 public 
schools, 123 Catholic schools, and 338 non-Catholic private schools. Of these, 138 schools (72 percent 
private) were later found to be ineligible because they did not have any kindergarten programs; three 
schools participated in fall-kindergarten, but not in spring-kindergarten (1 public and 2 private); 259 
schools (38 percent private) refused to participate in both fall and spring; and 65 schools (42 percent 
private) refused to participate in the fall, but were converted to cooperating schools in the spring during 
the spring refusal conversion. At the end of the base year, 1,014 schools were still participating in the 
ECLS-K. 



3.2.3 Sampling Children, Parents, and Teachers Within Schools 

The goal of the student sample design was to obtain an approximately self-weighting sample 
of students to the extent possible while achieving the minimum required sample size for APIs (the only 
subgroup that needed to be oversampled to meet the study’s precision goals). Two independent sampling 
strata were formed within each school, one containing API students and the second, all other students. 
Within each stratum, students were selected using equal probability systematic sampling, using a higher 
rate for the API stratum. In general, the target number of children sampled at any one school was 24. The 
actual sample size per school ranged from 1 to 28. If one twin was selected into the sample then both 
twins were included, raising the maximum number of children to sample from 24 to 28 in a small number 
of schools. Once the sampled children were identified, parent contact information was obtained from the 
school and was used to identify a parent or guardian for the parent interview. 

During the fall-kindergarten data collection, a census of kindergarten teachers was taken at 
each school. In spring-kindergarten, new teachers who had joined the schools and teachers in schools 
participating after the fall were added to the census of teachers. In the spring-first and spring-third grade 
data collections, the only teachers included were the teachers of the sampled children. For every data 
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collection, each sampled child was linked to his or her teacher. A child could be linked to only one 
general education teacher. In cases where a child had more than one general education teacher, a 'primary' 
teacher was identified for the child. In addition, special education teachers and service providers were 
linked to sample cases who received such services. As with the general education teachers, a child would 
be linked to only one special education teacher or service provider. Details on the linking of teachers to 
the children is found in Chapter 4. 



3.3 Fall-First Grade Subsample 

The fall data collection consisted of a 30 percent sample of schools containing 
approximately 25 percent of the base year students eligible to participate in the second year. The goal of 
this subsample was to measure the extent of summer learning loss and the factors that contribute to such 
loss and to better disentangle school and home effects on children’s learning 



3.3,1 PSU Sample 

A subsample of ECLS-K PSUs was selected for the fall-first grade data collection. All 24 of 
the self-representing PSUs were retained. Of the 76 non-self-representing PSUs, 38 were retained by 
sampling one PSU per stratum with equal probability. 



3.3.2 School Sample 

Base year schools in the 62 fall-first grade sampled PSUs were stratified by frame source 
(original public, original private, new from Catholic dioceses, new from local governments, etc.) and 
arranged in their original selection order. A 30 percent equal probability sample of schools was drawn in 
the 24 self-representing PSUs and a 60 percent sample of schools was drawn in the 38 non-self- 
representing PSUs. In total 311 schools that had cooperated in either fall- or spring-kindergarten were 
selected. The characteristics of the base year cooperating schools selected for fall-first grade are presented 
in table 3-10. 
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Table 3-10. Characteristics of base year cooperating schools selected for fall-first 
grade: School year 1999-2000 



Characteristic 


Total 


Public 


Private 


Total 


311 


228 


83 


Region 


Northeast 


57 


39 


18 


Midwest 


83 


59 


24 


South 


99 


77 


22 


West 


72 


53 


19 


Type of locale 


Large city 


62 


42 


20 


Midsize city 


59 


45 


14 


Urban fringe of large city 


86 


61 


25 


Urban fringe of midsize 


18 


14 


4 


city 


15 


12 


3 


Large town 


28 


19 


9 


Small town 


43 


35 


8 


Rural 

Religious affiliation (private 
schools only) 


Catholic 


29 


t 


29 


Other religious 


33 


t 


33 


Nonreligious, private 


21 


t 


21 


School type 


Regular 


292 


222 


70 


Ungraded 


1 


1 


0 


No grade beyond 


18 


5 


13 


kindergarten 



t Not applicable. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood 
Longitudinal Study, Kindergarten Class of 1998-99 kindergarten data collection, school year 2001-02. 



3.3.3 Child Sample 

Fall-first grade data collection consisted of the direct child assessment and the parent 
interview. Data collection was attempted for every eligible child found still attending the school in which 
he or she had been sampled during kindergarten. “Eligible” was defined as a base year respondent (i.e., a 
child who had either a fall- or spring-kindergarten child assessment or parent interview, or a child who 
could not be assessed because of a disability). Base year nonrespondents were not sampled and were 
handled by adjusting the weights (see chapter 7 for more details). 

Because of the additional burden of school recruiting, the cost of collecting data for a child 
who transferred from the school in which he or she was originally sampled exceeds that for a child who 
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stayed enrolled. To contain these costs, a random 50 percent of children were subsampled to be followed 
for fall-first grade data collection in the event that they had transferred. 

Prior to subsampling with equal probability, children were stratified into groups of 
nonmovers, movers with information identifying their new schools, and movers without such identifying 
information. A flag was created for each child indicating whether the child had been sampled to be 
followed. 



Except for children who were repeating kindergarten, all base year children sampled in 
schools with a high grade of kindergarten are de facto movers. Since many of these movers might move 
en masse to the same first grade school, steps were taken to follow these children at a higher rate. Using 
the information collected during spring-kindergarten, a list of destination schools was compiled for each 
such school. The destination school having the most movers was designated as primary, unless no such 
school had more than three movers. Children who moved en masse into a primary destination school in 
fall-first grade were treated as “nonmovers” and were not subsampled. All other movers were sampled at 
the rate of 50 percent. 



3.4 Spring-First Grade Sample 

The ECLS-K spring-first grade data collection targeted all base year respondents and not just 
the fall-first grade subsample. Hence, the sample includes children who were assessed and whose parents 
were interviewed in fall- or spring-kindergarten, as well as children who were excluded from the direct 
assessments because of a disability. In addition the spring student sample was freshened to include current 
first graders who had not been enrolled in kindergarten in 1998-99 and, therefore, had no chance of being 
included in the ECES-K base year kindergarten sample. This group includes children who skipped 
kindergarten altogether in 1998-99, children who attended a kindergarten program outside of the U.S. in 
1998-99, and children who were in first grade in 1998-99 and repeating it in 1999-2000. While all 
students still enrolled in their base year schools were recontacted, only a 50 percent subsample of base 
year sampled students who had transferred from their kindergarten school was followed for data 
collection. 
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3.4,1 



Subsampling Movers 



In spring-first grade all children in a random 50 percent subsample of base year schools were 
flagged to be followed for data collection if they transferred from their base year school. (This is in 
contrast to fall-first grade where a random 50 percent of children in each of the 30 percent of schools 
subsampled were flagged.) In order to maximize the amount of longitudinal data, care was taken during 
spring-first grade sampling to ensure that any child who had been flagged to be followed in fall-first grade 
would continue to be followed. 

In selecting the spring-first grade 50 percent subsample of schools where movers would be 
flagged for followup, the three primary strata were self-representing PSUs, non-self-representing PSUs 
that had been selected for fall-first grade, and non-self-representing PSUs that had not been selected for 
fall-first grade. Within these major strata, schools were grouped by frame source (original public, original 
private, new from Catholic dioceses, new from local governments, etc.). Finally within each frame source, 
schools were stratified by cooperation status (i.e., cooperating in spring-kindergarten or not cooperating in 
spring-kindergarten), and arranged in this original selection order. Schools that had been part of the 30 
percent fall-first grade sample were automatically retained. Then equal probability sampling methods 
were employed to augment the sample to the desired 50 percent of schools. The net result of these 
procedures was that every base year selected school had a 50 percent chance of having its ECLS-K 
movers followed during spring-first grade, and any mover who had been followed in fall-first grade 
would still be followed in spring-first grade. 



3.4.2 Sample Freshening 

As noted earlier, a sample freshening procedure was used to make it possible to produce 
estimates of all children enrolled in first grade in the spring of 2000. The spring-first grade student 
freshening used a half-open interval sampling procedure (Kish, 1965). The procedure was implemented in 
the same 50 percent subsample of ECLS-K base year schools where movers were flagged for followup. 
Each of these schools was asked to prepare an alphabetic roster of students enrolled in first grade and the 
names of ECLS-K kindergarten-sampled students were identified on this list. Beginning with the name of 
the ECLS-K first kindergarten-sampled child, school records were checked to see whether the student 
directly below in the sorted list attended kindergarten in the United States in fall 1998. If not, (1) that 
child was considered to be part of the freshened sample and was linked to the base year sampled student 
(i.e., was assigned that student’s probability of selection), and (2) the record search procedure was 
repeated for the next listed child, and so forth. When the record search revealed that a child had been 
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enrolled in kindergarten the previous year, that child was not considered part of the freshened sample and 
the procedure was started all over again with the second base year ECLS-K sampled student name, and so 
on.* Student freshening brought 165 first graders into the ECLS-K sample, which increased the weighted 
survey estimate of the number of first graders in the United States by about 2.6 percent. 

The student freshening procedure was not entirely free of bias. A first grader would have no 
chance of being in the ECLS-K first grade sample if he or she was enrolled in a school where neither the 
child nor any of his or her classmates had attended kindergarten in the United States in fall 1998. This 
would be a rare circumstance and is not thought to be an important source of bias. A more significant 
source of potential bias is nonresponse. One source of nonresponse inherent to the freshening plan was 
that the procedure only involved students who had not transferred from the school in which they had been 
sampled during the base year. Another source of nonresponse that also affected the freshening procedure 
was schools that refused to provide or could not provide the necessary information such as alphabetic 
roster of students enrolled in first grade, or whether students attended kindergarten the previous year. 



3.5 Spring-Third Grade Sample Design Issues 

The procedures used in spring-first grade to subsample movers reduced the loss in sample 
size and reduced data collection costs since movers cost considerably more to interview than 
nonmovers. These procedures were also used for the ECLS-K third grade data collection with some 
modifications. One reason for modifying the procedures was that some children had already moved out of 
their original school, and some of the movers were sampled and some were not. In addition, there were 
concerns about special domains of interest and methods that might be used to increase the sample size for 
the children in these groups. Results from the first grade collection were used to address these third grade 
sample design issues. 



3.5.1 Estimates from Spring-First Grade 

Table 3-11 presents data on the outcome of the spring-first grade data collection activities, 
by subgroups of interest and by mover status. In this table and subsequent tables, school type and 
urbanicity are those of the original sample schools. Race/ethnicity and language characteristics of the 
children are from the ECLS-K base year data, which are available for all children. Data from first grade 

* The student roster was “circularized” (i.e., the first name on the roster was considered to follow the last name on the roster in the 
implementation of the procedure). 
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are only available for first grade respondents. For children sampled in first grade, data are from spring- 
first grade. The table shows that overall 26 percent (5,477) of the children moved from the school they 
were sampled in, about 48 percent (2,620) of these movers were sampled, and the unweighted response 
rate for movers was 83 percent (1,967 mover respondents). For nonmovers, the response rate was 97 
percent (15,357 nonmover respondents). A child was considered a respondent in these computations if 
either the child assessment or the parent interview was completed for spring-first grade or the child was 
not assessed due to a disability. 
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Table 3-11. Spring-first grade data collection results by mover status: School year 2001-02 



Mover status 



Subgroup 



All 

School type 

Public 

Private 

Catholic 

Non-Catholic 

Urbanicity 

Rural 

Non-rural 

Race/ethnicity 1^ 

Hispanic 

Black 

Asian/Pacific 

Islander 

Other 

Race/ethnicity 2 ^ 

Hispanic 

Black 

Asian/Pacific 

Islander 

Other 

Language minority 

Non-English 

English 



All sampled children Movers Nonmovers 



Total' 


Respond^ 


Not 

respond Ineligible 


Unweighted 
resp. rate 


Movers not 
sampled 


Total Percent 
movers moved 


Respond^ 


Not Percent Unweighted 

respond Ineligible sampled resp. rate 


Respond^ 


Not 

respond Ineligible 


Unweighted 
resp. rate 


21,331 


17,324 


899 


251 


95.1 


2,857 


5,477 


25.7 


1,967 


403 


250 


47.8 


83.0 


15,357 


496 


1 


96.9 


16,761 


13,661 


710 


221 


95.1 


2,169 


4,189 


25.0 


1,466 


334 


220 


48.2 


81.4 


12,195 


376 


1 


97.0 


4,570 


3,663 


189 


30 


95.1 


688 


1,288 


28.2 


501 


69 


30 


46.6 


87.9 


3,162 


120 


0 


96.3 


2,354 


2,031 


66 


12 


96.9 


245 


433 


18.4 


150 


26 


12 


43.4 


85.2 


1,881 


40 


0 


97.9 


2,216 


1,632 


123 


18 


93.0 


443 


855 


38.6 


351 


43 


18 


48.2 


89.1 


1,281 


80 


0 


94.1 


2,509 


2,227 


86 


14 


96.3 


182 


428 


17.1 


194 


39 


13 


57.5 


83.3 


2,033 


47 


1 


97.7 


18,822 


15,097 


813 


237 


94.9 


2,675 


5,049 


26.8 


1,773 


364 


237 


47.0 


83.0 


13,324 


449 


0 


96.7 


3,777 


2,988 


164 


81 


94.8 


544 


1,048 


27.7 


349 


75 


80 


48.1 


82.3 


2,639 


89 


1 


96.7 


3,229 


2,468 


150 


69 


94.3 


542 


1,066 


33.0 


374 


81 


69 


49.2 


82.2 


2,094 


69 


0 


96.8 


1,579 


1,291 


61 


18 


95.5 


209 


359 


22.7 


114 


18 


18 


41.8 


86.4 


1,177 


43 


0 


96.5 


12,746 


10,577 


524 


83 


95.3 


1,562 


3,004 


23.6 


1,130 


229 


83 


48.0 


83.1 


9,447 


295 


0 


97.0 


3,698 


2,926 


160 


81 


94.8 


531 


1,023 


27.7 


340 


72 


80 


48.1 


82.5 


2,586 


88 


1 


96.7 


3,229 


2,468 


150 


69 


94.3 


542 


1,066 


33.0 


374 


81 


69 


49.2 


82.2 


2,094 


69 


0 


96.8 


1,867 


1,537 


69 


20 


95.7 


241 


429 


23.0 


144 


24 


20 


43.8 


85.7 


1,393 


45 


0 


96.9 


12,537 


10,393 


520 


81 


95.2 


1,543 


2,959 


23.6 


1,109 


226 


81 


47.9 


83.1 


9,284 


294 


0 


96.9 


5,372 


4,317 


228 


107 


95.0 


720 


1,397 


26.0 


472 


98 


107 


48.5 


82.8 


3,845 


130 


0 


96.7 


15,959 


13,007 


671 


144 


95.1 


2,137 


4,080 


25.6 


1,495 


305 


143 


47.6 


83.1 


11,512 


366 


1 


96.9 



'The total number of children excludes 68 children who responded in fall-kindergarten and became ineligible in spring-kindergarten, and includes 139 children sampled in first grade who responded. 

^A respondent is a child with assessment data or parent interview data, or a child who could not be assessed due to a disability. 

^Race/ethnicity 1 was the strict definition of API (RACE=5-Asian, or 6-Native Hawaiian or other Pacific Islander), while race/ethnicity 2 was the broader definition (RACE=5-Asian, or 6-Native Hawaiian or other Pacific Islander 
or WKAS1AN=1 -Child is Asian, or WKPACISL=1-Child is Native Hawaiian or other Pacific Islander). Variables are from the ECLS-K base year data file. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten, first, and third grade data collections, school years 1998-99, 
1999-2000, and 2001-02. 



The mover rates show the types of variation that had been expected, with higher mover rates 
for Black and Hispanic children, for example. A total of 39 percent of the children in non-Catholic private 
moved to public schools when they advanced from kindergarten to first grade. Seventy-six percent of 
children who moved from kindergarten in private schools to first grade in public schools attended non- 
Catholic private schools in kindergarten, about three times the number of children who moved from 
Catholic to public schools. 

One of the concerns in using the kindergarten to first grade mover rates to make estimates 
for future transitions was whether the mover rates for the 1-year time period between kindergarten and 
first grade were reasonable when applied to the transition between first and third grade. One might argue 
that a 2-year period should result in a higher mover rate than the 1-year rate. However, parents may be 
more reluctant to change the school for a child between first and third grade than between kindergarten 
and first grade. Kindergarten is also special for other reasons, for example, the availability of full- and 
part-day classes may be an important factor in the choice of the kindergarten. There are no other data 
sources that could be used to examine differential mover rates between years. As a result, the 1-year 
moving rates in table 3-11 were applied to the 2-year period between first and third grade after adding 
another 5 percent to the rates to account for the 2-year period. An exception was made for children who 
attended non-Catholic private schools in the base year and had the highest rates of moving among all the 
domains examined. This was assumed to be a special case for kindergarten and the average mover 
percentage was applied to these children for the third grade. 

The other main concern was whether the extremely high response rate for nonmovers (97 
percent) could be duplicated in future years. To be more conservative and to account for the fact that 
nonrespondents from earlier rounds (i.e., base year respondents were included in the third grade sample 
whether or not they responded in first grade) were included in subsequent rounds of data collection, it was 
assumed that a 95 percent response rate would be achieved for nonmovers in third grade. 



3.5.2 Third Grade Sample Design 

The basic plan for third grade was the plan adopted for first grade where only 50 percent of 
the children who moved from the original sample schools were followed into their new school. This plan 
was modified for third grade as described below. The basic plan was to include all eligible children in the 
third grade sample if they remained in the school they attended previously and to treat movers according 
to the following specifications: 
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■ To be eligible for the sample, a child had to have been a base year respondent or 
sampled in first grade. Children who moved out of the country or died were excluded 
(i.e., ineligible). 

■ All the respondents in the base year who remained in their original schools, where the 
original schools also included destination schools (described later). 

■ All the children who moved from an original school in a previous wave of data 
collection and were retained in the subsample of movers for that wave. For example, 
if a child moved between kindergarten and first grade and was part of the 50 percent 
subsample that was followed, then the child would be retained for future rounds 
without subsampling as long as the child remained eligible. 

■ A subsample of 50 percent of the children who moved from their original school at 
any time after the base year. For example, a child who moved between first grade and 
third grade would be subject to subsampling and had a 50 percent chance of being 
included in the third grade followup. In alternatives discussed later, differential 
subsampling rates were introduced. 

To prevent an accumulation of nonresponse, the ECLS-K design does not use the approach 
of many longitudinal studies that excludes sampled units from future rounds if they did not respond in a 
particular wave. Instead, the basic plan was modified so that all eligible base-year respondents who were 
sampled in the first grade followup would be eligible for the third grade followup even if they did not 
respond in the first grade. Even though the participation rate for first grade nonrespondents might be 
lower compared with first grade respondents in the subsequent followups, the effort was an attempt to 
increase overall response rates by including first grade nonrespondents in third grade. The approach is 
also consistent with the analytic use of the data for the ECES-K, since many analyses may include less 
than complete wave responses. For example, a change in scores from kindergarten to third grade for 
subgroups is an important analytic objective, and it can be estimated without complete data at each wave. 



A second procedure that was part of the modification of the basic plan for the third grade 
followup was an extension of a procedure that was used in the first grade followup to deal with schools 
that ended with kindergarten (i.e., kindergarten was the highest grade offered). A school was called a 
destination school if at least 4 students from a school ending in kindergarten attended this school in first 
grade. For the third grade 28 original schools ended in second grade, and 3 of the destination schools 
identified in first grade ended in second grade. In total, 3 percent of all eligible first graders in the ECES- 
K sample attended schools ending in second grade. As was done for the first grade sample, children in the 
destination schools were treated as nonmovers for the third grade sample. As nonmovers, they were all 
followed into their new schools resulting in a 2 percent increase of the third grade sample size over that 
which would result if 50 percent of these children were subsampled out as movers. 



3-29 




3.5.3 



Expected Sample Size 



Table 3-12 gives the expeeted sample sizes of children in third grade by subgroups of 
interest and mover status for the basic plan. In this table, a respondent is defined as a child with either a 
complete child assessment or parent interview, or a child who could not be assessed because of a 
disability. This table shows that the expected number of children with completed child assessments or 
parent interviews in the third grade is 14,304 under the assumed mover rates (differential by subgroups as 
shown in the table), subsampling rate (47 percent instead of 50 percent to account for ineligibility in third 
grade), and response rate (95 percent). The estimates for the selected groups of high interest are given in 
the rows below. The third from last column is the estimated design effect resulting from sampling movers 
and nonmovers differentially. It does not include any other factors such as clustering. The next to last 
column is an estimate of the number of new schools that would enter the ECLS-K sample. The last 
column is an estimate of the total number of schools that sampled children would be attending, assuming 
1.5 movers attended the same school on average. 



3.5.4 Protecting the Language Minority Children 

Special attention was paid to language minority and API children to ensure that the sample 
sizes would be large enough to support analytic goals in developing the sampling plans for the third 
grade. Children in the language minority group are children whose home language was non-English or 
who were screened using the Oral Eanguage Development Scale (OEDS) prior to assessments during the 
base year (or first grade for freshened children).® Two classifications of APIs are shown in table 3-11. The 
first classification was identified using a strict definition of API, i.e., if the child was identified only as 
API by the composite race variable (RACE = 5-Asian or 6-Native Hawaiian or other Pacific Islander). 
The second classification was identified using a broader definition, i.e., if a child was identified only as 
API as in the strict definition (RACE = 5-Asian or 6-Native Hawaiian or other Pacific Islander) or if a 
child has positive answers to the API race identification variables (WKASIAN = 1 -Child is Asian or 
WKPACISE = 1 -Child is Native Hawaiian or other Pacific Islander). The variables RACE, WKASIAN 
and WKPACISE are in the base year data file. The broader definition of API yields a larger population of 
children. 



’ Information about home language came from the parent interview and whether or not children were screened with the OLDS was based on 
information provided by their schools (see the ECLS-K Base Year Public-Use Data Files and Electronic Code Book: User ’s Manual). 
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Table 3-12. Expected sample size for selected subgroups for third grade, by mover status: School year 2001-02 



Subgroup 



Sampled from first grade 



Total Movers Nonmovers 



Mover Response rates Sample of Total responding (expected) Subsample New third grade Total 

Rate Sampled^ Movers Nonmovers new movers Total Movers Nonmovers design effect schools^ schools 



All children 


18,223 


2,370 


15,853 


— 


— 


— 


— 


2,284 


14,304 


3,860 


10,444 


t 


1,523 


4,144’ 


School type 


Public 


14,371 


1,800 


12,571 


30 


47 


81 


95 


1,772 


11,270 


2,909 


8,361 


1.17 


1,182 


— 


Private 


3,852 


570 


3,282 


33 


47 


88 


95 


512 


3,034 


951 


2,083 


1.14 


341 


— 


Catholic 


2,097 


176 


1,921 


23 


47 


85 


95 


211 


1,728 


330 


1,398 


1.13 


— 


— 


Non-Catholic 


1,755 


394 


1,361 


31 


47 


89 


95 


198 


1,420 


528 


892 


1.27 


— 


— 


Urbanicity 


Rural 


2,313 


233 


2,080 


22 


47 


83 


95 


216 


1,914 


374 


1,540 


1.09 


— 


— 


Non-rural 


15,910 


2,137 


13,773 


32 


47 


83 


95 


2,080 


12,402 


3,482 


8,920 


1.17 


— 


— 


Race/ethnicity 1“^ 


Hispanic 


3,152 


424 


2,728 


33 


47 


82 


95 


420 


2,438 


695 


1,743 


1.18 


t 


t 


Black 


2,618 


455 


2,163 


38 


47 


82 


95 


386 


1,965 


692 


1,274 


1.17 


t 


t 


Asian/Pacific 


Islander 


1,352 


132 


1,220 


28 


47 


86 


95 


159 


1,089 


251 


838 


1.17 


t 


t 


Other 


11,101 


1,359 


9,742 


29 


47 


83 


95 


1,308 


8,829 


2,218 


6,611 


1.14 


t 


t 


Race/ethnicity 2'^ 


Hispanic 


3,086 


412 


2,674 


33 


47 


83 


95 


411 


2,389 


679 


1,711 


1.18 


t 


t 


Black 


2,618 


455 


2,163 


38 


47 


82 


95 


386 


1,965 


692 


1,274 


1.17 


t 


t 


Asian/Pacific 


Islander 


1,606 


168 


1,438 


28 


47 


86 


95 


189 


1,290 


306 


984 


1.16 


t 


t 


Other 


10,913 


1,335 


9,578 


29 


47 


83 


95 


1,288 


8,675 


2,179 


6,497 


1.15 


t 


t 


Language minority 


Non-English 


4,545 


570 


3,975 


31 


47 


83 


95 


579 


3,557 


952 


2,605 


1.17 


t 


t 


English 


13,678 


1,800 


11,878 


31 


47 


83 


95 


1,706 


10,747 


2,912 


7,835 


1.15 


t 


t 



— Not available, 
t Not applicable. 

' The sampling rate for movers is set at 47 percent (instead of 50 percent) to account for ineligibility of students in future rounds. 

^ The number of new third grade schools is estimated as 1 .5 schools per sampled new mover. 

^ 2,62 1 schools from kindergarten/first grade plus the new third grade schools. 

Race/ethnicity 1 was the strict definition of API (RACE=5-Asian or 6-Native Hawaiian or other Pacific Islander), while race/ethnicity 2 was the broader definition (RACE=5-Asian or 6-Native Hawaiian or other Pacific Islander or 
WKASIAN=1-Child is Asian or WKPACISL=1-Child is Native Hawaiian or other Pacific Islander). Variables are from the ECLS-K. base year data file. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten, first, and third grade data collections, school years 1998-99, 
1999-2000, and 2001-02. 



After reviewing the expected yields without oversampling, it was decided to increase only 
the sample size for children belonging to the language minority group. Beginning in third grade, these 
children would not be subsampled for followup if they moved from their original school. Instead, data 
collection would be attempted for all language minority children. Table 3-13 is analogous to table 3-12 
but is adjusted for this approach of retaining all movers in the language minority group (in practice the 
subsampling rates are shown as 95 percent because some children became ineligible). 

One consequence of protecting this subgroup is to increase the sample size and precision for 
the subgroup. The design effect due to subsampling is slightly lower under this plan because a smaller 
proportion of the movers were subsampled than under the basic plan (only the movers that were already 
subsampled in first grade are subsampled). Another consequence is that the number of schools that the 
sampled children attended increased. Because all language minority children were followed, table 3-13 
shows an expected increase of 395 schools in third grade (1,918 - 1,523 = 395). 



3.5.5 Cost Savings Options 



As mentioned earlier, the cost of collecting data for a child who transferred from the school 
in which he or she was originally sampled exceeds that for a child who stayed enrolled at the same school. 
The additional costs include recruiting the schools and collecting data within a school for one or very few 
children. This differential cost is the primary reason that all movers were not followed in the ECLS-K. 
For example, base year respondents attended 1,014 schools in the base year. The number of schools with 
at least one sampled child after subsampling movers increased to 2,131 by spring-first grade and to 3,225 
by spring-third grade. The following options were considered to reduce the cost of the data collection, but 
none were implemented in third grade. 



Collect only child assessment and parent interview data for movers. This approach 
would eliminate some of the costs of contacting school districts and schools for a 
child in a school where there were no other sampled children 

Drop movers from the sample if they did not respond in a previous wave (defined by 
no child or parent data from the wave). 

Subsample all movers or some set of movers (such as those that moved between 
waves 1 and 4) at a lower rate than 50 percent. 
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Table 3-13. Preserving the language minority children in third grade: School year 2001-02 



Sampling approach 


Sampled from first grade 




Mover 


Response rates 




Sample of 


Total responding (expected) 


Subsample 


New third 


Total 


Total 


Movers 


Nonmovers 


Rate 


Sampled* 


Movers Nonmovers 


new movers 


Movers 


Nonmovers 


Total 


design effect 


grade schools^ 


schools^ 


Using standard 
subsampling rates 


18,223 


2,370 


15,853 


t 


t 


t 


t 


2,285 


3,864 


10,440 


14,304 


t 


1,523 


4,144 


Non-English 


4,545 


570 


3,975 


31 


47 


83 


95 


579 


952 


2,605 


3,557 


1.17 


t 


t 


English 


13,678 


1,800 


11,878 


31 


47 


83 


95 


1,706 


2,912 


7,835 


10,747 


1.15 


t 


t 


Preserving the 
language 
minority group 


18,223 


2,370 


15,853 


t 


t 


t 


t 


2,877 


4,354 


10,440 


14,794 


t 


1,918 


4,539 


Non-English 


4,545 


570 


3,975 


31 


95 


83 


95 


1,171 


1,442 


2,605 


4,047 


1.04 


t 


t 


English 


13,678 


1,800 


11,878 


31 


47 


83 


95 


1,706 


2,912 


7,835 


10,747 


1.15 


t 


t 



t Not applicable. 

* The sampling rate for movers is set at 47 percent (instead of 50 percent) to account for ineligibility of students in future rounds. If the language minority group is preserved, it is set at 95 percent. 

^ The number of new third grade schools is estimated as 1 .5 schools per sampled new mover. 

^ 2,621 schools from kindergarten/first grade plus the new third grade schools. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten, first, and third grade data 
collections, school years 1998-99, 1999-2000, and 2001-02. 



Specify a minimum response set of data for children across rounds of data collection 
and subsample or eliminate children who do not meet this minimum. 



The main disadvantage of the last three options is that the sample size would be reduced, 
thus jeopardizing analyses of data for smaller subgroups. Dropping data collection of schools and teachers 
as described in the first option would eliminate any longitudinal analyses using school and teacher data. 



3.5.6 Precision Requirements 

When the precision estimates were computed from the kindergarten sample at the end of the 
base year, higher than expected design effects for assessment scores were observed. The design effects for 
most other statistics, such as proportions of children with a particular characteristic, were moderate and 
within the range expected (1.6 to 6.9 for proportions greater than 30 percent for an average of 4.0). The 
design effects for assessment scores (4.5 to 9.5 for an average of 6.9) were investigated and found to be 
correct and unrelated to data collection artifacts. For example, interviewer effects were found to be 
negligible and did not bias assessment scores. The design effects for test scores were much larger than the 
average of 3.8 that was expected at the design stage. For all students, the design effects for math and 
reading scores were about 6.5, while for general knowledge the design effects were even larger at 7.7. For 
design effects from the base year, see chapter 4 of the Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 (ECLS-K), Base Year Public-Use Data Files and Electronic Code Book: User’s Manual 
(NCES 2001-029: U.S. Department of Education, National Center for Education Statistics, 2001). These 
larger design effects are one component that affects the ability of the survey to meet the precision 
requirements as described in section 3.1. 

The spring-third grade estimates of design effects are similar to those in the earlier rounds 
and are larger than had been predicted prior to any data collection. The longitudinal estimates have design 
effects that are not as large as might be expected given the larger cross-sectional design effects. In fact, 
the correlations for mean test scores seem to be running as high as .8 to .9. The higher than expected 
correlations make it possible to meet the precision requirements for estimates of change with smaller 
sample sizes. Table 3-14 shows that the sample sizes for most of the key analytic subgroups were 
expected to be approximately 1,000. Samples of this size were expected to be sufficient for estimating 
most characteristics. For example, test scores were expected to have a coefficient of variation of about 3 
percent with samples of 1,000. More details on estimates of design effects can be found in chapter 4 of 
the ECLS-K User’s Manual for the First Grade Public-Use Data Files and Electronic Code Book (NCES 
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2002-135; U.S. Department of Education, National Center for Education Statistics, 2002) and the ECLS-K 
User’s Manual for the Third Grade Public-Use Data File and Electronic Code Book (NCES 2004-001; 
U.S. Department of Education, National Center for Education Statistics, forthcoming). 



Table 3-14. Characteristics of third grade respondents — number of third graders by subgroup: 
School year 2001-02 







Expected* 






Achieved 




Subgroup 


Total 


Eanguage 

minority 


Not 

language 

minority 


Total 


Eanguage 

minority 


Not 

language 

minority 


Total 


14,794 


4,047 


10,747 


15,305 


4,041 


11,264 


School type 


Public 


11,643 


3,356 


8,287 


12,070 


3,374 


8,696 


Private 


3,151 


691 


2,460 


3,235 


667 


2,568 


Catholic 


1,625 


373 


1,252 


1,817 


383 


1,434 


Non-Catholic 


1,526 


318 


1,208 


1,418 


284 


1,134 


Urbanicity 


Rural 


1,712 


210 


1,502 


2,005 


222 


1,783 


N on-rural 


13,082 


3,837 


9,245 


13,300 


3,819 


9,481 


Race/ethnicity f 


Hispanic 


2,773 


2,165 


608 


2,752 


2,156 


596 


Black 


2,188 


128 


2,060 


2,007 


118 


1,889 


Asian/Pacific 


Islander 


1,165 


962 


203 


1,174 


947 


227 


Other 


8,668 


792 


7,876 


9,372 


820 


8,552 


Race/ethnicity 2^ 


Hispanic 


2,716 


2,130 


587 


2,691 


2,116 


575 


Black 


2,188 


128 


2,060 


2,007 


118 


1,889 


Asian/Pacific 


Islander 


1,369 


1,057 


312 


1,404 


1,056 


348 


Other 


8,520 


732 


7,788 


9,203 


751 


8,452 



' The expected sample size was computed using assumed mover rates (differential by subgroups as shown table 3-12), a 47 percent 
subsampling rate (instead of 50 percent to account for ineligibility in third grade), and a 95 percent response rate. 

^ Race/ethnicity 1 was the strict definition of API (RACE=5-Asian, or 6-Native Hawaiian or other Pacific Islander), while race/ethnicity 2 
was the broader definition (RACE=5-Asian, or 6-Native Hawaiian or other Pacific Islander or WKASIAN=1 -Child is Asian, 
or WKPACISL=1 -Child is Native Hawaiian or other Pacific Islander). Variables are from the ECLS-K base year data file. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 third grade data collection, school year 2001-02. 



3-35 




3.6 



Spring-Third Grade Sample 



To summarize, the sample of children for spring-third grade consists of all children who 
were base year respondents and children who were brought into the sample in spring-first grade through 
the sample freshening procedure. Sample freshening was not implemented in third grade; hence no new 
students entered the sample. 

While all students still enrolled in their base year schools were recontacted, slightly more 
than 50 percent of the base year sampled students who had transferred from their kindergarten school 
were followed for data collection. This subsample of students was the same 50 percent subsample of base 
year movers followed in spring-first grade, including the movers whose home language was not English 
(language minority students). Children who were followed in spring-first grade were retained in the 
sample (i.e., the mover followup still targeted the same 50 percent subsample of children in the base year 
schools). In addition, children whose home language was not English and who moved between spring- 
first grade and spring-third grade were all retained rather than being subsampled at the 50 percent rate. If 
they had moved before first grade, they were not to be followed. This modification to the mover followup 
procedure provided a larger sample of children whose home language is not English for analytic purposes. 
The mover followup activities that originally targeted a 50 percent subsample of children in base year 
schools resulted in a 54 percent subsample with the addition of language minority children. 

Tables 3-14 (count) and 3-15 (percent) show the characteristics of the achieved third grade 
sample compared with the expected third grade sample. The total number of children in the language 
minority group is virtually the same as the expected number while the total number of children in the 
other group is about 5 percent larger than the expected number. In computing the expected sample size, 
the same mover rate was assumed for both groups of children. The third grade sample shows that the non- 
language minority children moved at a lower rate (42 percent) than the language minority children (44 
percent) resulting in a slightly larger sample of non-language minority children. The agreement between 
the expected and achieved sample sizes is rather remarkable given the numerous assumptions required. 
The actual percent distribution of third graders within each subgroup is as expected with the exception of 
the Catholic and non-Catholic private schools where the percent of children in Catholic schools is higher 
than that of children in non-Catholic private schools. This may be due to the lower completion rate of 
children in non-Catholic private schools compared with children in Catholic private schools (93 percent 
and 97 percent, respectively). Elsewhere among the children in the language minority group, the 
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difference between the expected distribution and the actual distribution is less than 1 percent. Elsewhere 
among the children not in the language minority group, the difference is less than 3 percent. 

Table 3-15. Characteristics of third grade respondents — percent distribution by subgroup: 

School year 2001-02 







Expected' 






Achieved 




Subgroup 


Total 


Eanguage 

minority 


Not 

language 

minority 


Total 


Eanguage 

minority 


Not 

language 

minority 


Total 


100.0 


100.0 


100.0 


100.0 


100.0 


100.0 


School type 


Public 


78.7 


82.9 


77.1 


78.9 


83.5 


77.2 


Private 


21.3 


17.1 


22.9 


21.1 


16.5 


22.8 


Catholic 


51.6 


54.0 


50.9 


56.2 


57.4 


55.8 


Non-Catholic 


48.4 


46.0 


49.1 


43.8 


42.6 


44.2 


Urbanicity 


Rural 


11.6 


5.2 


14.0 


13.1 


5.5 


15.8 


Non-rural 


88.4 


94.8 


86.0 


86.9 


94.5 


84.2 


Race/ethnicity l' 


Hispanic 


18.7 


53.5 


5.7 


18.0 


53.4 


5.3 


Black 


14.8 


3.2 


19.2 


13.1 


2.9 


16.8 


Asian/Pacific 


Islander 


7.9 


23.8 


1.9 


7.7 


23.4 


2.0 


Other 


58.6 


19.6 


73.3 


61.2 


20.3 


75.9 


Race/ethnicity 2' 


Hispanic 


18.4 


52.6 


5.5 


17.6 


52.4 


5.1 


Black 


14.8 


3.2 


19.2 


13.1 


2.9 


16.8 


Asian/Pacific 


Islander 


9.3 


6.1 


.9 


9.2 


26.1 


3.1 


Other 


57.6 


18.1 


72.5 


60.1 


18.6 


75.0 



'The expected sample size was computed using assumed mover rates (differential by subgroups as shown table 3-12), a 47 percent 
subsampling rate (instead of 50 percent to account for ineligibility in third grade), and a 95 percent response rate. 

^Race/ethnicity 1 was the strict definition of API (RACE=5-Asian, or 6-Native Hawaiian or other Pacific Islander), while 
race/ethnicity 2 was the broader definition (RACE=5-Asian, or 6-Native Hawaiian or other Pacific Islander, or WKASIAN=1 -Child is 
Asian, or WKPACISL=1 -Child is Native Hawaiian or other Pacific Islander)). Variables are from the ECLS-K base year data file. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 



Table 3-16 shows the third grade data collection results by mover status (analogous to table 
3-11). In this table, the total number of children is 21,357 which is larger than the total in table 3-11 by 26 
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Table 3-16. Spring-third grade data collection results by mover status: School year 2001-02 



Spring-third grade response status Movers-sampled Nonmovers 

Not Not Unweighted Not Percent Percent Unweighted Not Unweighted 





Total' 


Respond^ 


respond 


Ineligible 


sampled 


resp. rate 


Respond^ 


respond 


Ineligible 


moved 


sampled 


resp. rate 


Respond^ 


respond 


Ineligible 


resp. rate 


All 


21,357 


15,305 


1,524 


364 


4,164 


90.9 


3,583 


931 


361 


42.3 


53.9 


79.4 


11,722 


593 


3 


95.2 


School type 


































Public 


16,784 


12,070 


1,236 


312 


3,166 


90.7 


2,726 


760 


311 


41.5 


54.5 


78.2 


9,344 


476 


1 


95.2 


Private 


4,573 


3,235 


288 


52 


998 


91.8 


857 


171 


50 


45.4 


51.9 


83.4 


2,378 


117 


2 


95.3 


Catholic 


2,354 


1,817 


113 


14 


410 


94.1 


329 


62 


14 


34.6 


49.7 


84.1 


1,488 


51 


0 


96.7 


Non-Catholic 


2,216 


1,418 


172 


38 


588 


89.2 


528 


109 


36 


56.9 


53.4 


82.9 


890 


63 


2 


93.4 


Urbanicity 


































Rural 


2,478 


2,005 


162 


20 


291 


92.5 


304 


102 


19 


28.9 


59.4 


74.9 


1,701 


60 


1 


96.6 


Non-rural 


18,879 


13,300 


1,362 


344 


3,873 


90.7 


3,279 


829 


342 


44.1 


53.5 


79.8 


10,021 


533 


2 


94.9 


Race/ethnicity 1^ 


































Hispanic 


3,777 


2,752 


274 


145 


606 


90.9 


756 


193 


144 


45.0 


64.3 


79.7 


1,996 


81 


1 


96.1 


Black 


3,229 


2,007 


267 


81 


874 


88.3 


604 


186 


81 


54.0 


49.9 


76.5 


1,403 


81 


0 


94.5 


Asian/Pacific 
























76.4 


876 


47 


0 


94.9 


Islander 


1,579 


1,174 


139 


37 


229 


89.4 


298 


92 


37 


41.5 


65.1 












Other 


12,772 


9,372 


844 


101 


2,455 


91.7 


1,925 


460 


99 


38.7 


50.3 


80.7 


7,447 


384 


2 


95.1 


Race/ethnicity 2 ^ 


































Hispanic 


3,698 


2,691 


271 


144 


592 


90.9 


739 


191 


143 


45.0 


64.4 


79.5 


1,952 


80 


1 


96.1 


Black 

Asian/Pacific 


3,229 


2,007 


267 


81 


874 


88.3 


604 


186 


81 


54.0 


49.9 


76.5 


1,403 


81 


0 


94.5 


Islander 


1,867 


1,404 


151 


40 


272 


90.3 


351 


102 


40 


41.0 


64.4 


77.5 


1,053 


49 


0 


95.6 


Other 


12,563 


9,203 


835 


99 


2,426 


91.7 


1,889 


452 


97 


38.7 


50.1 


80.7 


7,314 


383 


2 


95.0 


Language minority 


































Non-English 


5,372 


4,041 


412 


203 


716 


90.7 


1,162 


291 


203 


44.2 


69.8 


80.0 


2,879 


121 


0 


96.0 


English 


15,985 


11,264 


1,112 


161 


3,448 


91.0 


2,421 


640 


158 


41.7 


48.3 


79.1 


8,843 


472 


3 


94.9 



*The total number of children excludes 68 children who responded in fall-kindergarten and became ineligible in spring-kindergarten, and includes 165 children sampled in first grade who were eligible. 

respondent is a child with assessment data or parent interview data, or a child who could not be assessed due to a disability. 

^ Race/ethnicity 1 was the strict definition of API (RACE=5-Asian or 6-Native Hawaiian or other Pacific Islander), while race/ethnicity 2 was the broader definition (RACE=5-Asian or 6-Native 
Hawaiian or other Pacific Islander or WKASIAN=1 -Child is Asian or WKPACISL=1 -Child is Native Hawaiian or other Pacific Islander). Variables are from the ECLS-K base year data file. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 kindergarten, first, and third grade data 
collections. 



children; these are children sampled in first grade who did not have completed assessment data or parent 
interview in first grade (hence not included in table 3-11), but participated in the third grade study (hence 
included in table 3-16). Overall the unweighted response rate for third grade is 79 percent for movers and 
95 percent for nonmovers, compared with the expected response rate of 83 and 95 percent. The rate of 
base year respondents who moved out of their original sample schools is 42 percent (compared with the 
expected overall moving rate of 47 percent). Note that in all tables in this chapter a respondent is defined 
as a child with completed assessment data or completed parent interview data or a child who could not be 
assessed due to a disability, so that the response rate calculated here is not the same as the completion rate 
in chapter 5 of the ECLS-K User’s Manual for the Third Grade Public-Use Data File and Electronic 
Code Book (NCES 2004-001) which is instrument-specific. 

As seen in table 3-1, the achieved sample for third grade is consistent with the goal to 
achieve a fifth grade sample of 10,300 children with completed data. By the end of third grade, there were 
14,470 children with complete assessment data or not assessed due to a disability; 13,489 children with 
completed parent data; and 15,305 children in either group. There were 12,654 children with the more 
restricted condition, i.e., with complete assessment data (or not assessed due to a disability) and parent 
data. These sample sizes suggest that the fifth grade target sample size can be achieved. 
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4. DATA COLLECTION METHODS 



The following sections discuss the data collection procedures in the third grade data 
collection phase of the Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K). 
Section 4. 1 gives an overview of the data collection methods. Detailed information is provided on roles 
and responsibilities in the study (section 4.2), study training procedures (section 4.3), fall preassessment 
school contacts (section 4.4), spring-third grade data collection (section 4.5), and data collection quality 
control procedures (section 4.6). 



4.1 Overview of Data Collection Methods 

The ECES-K third grade data collection was conducted in the fall and spring of the 2001-02 
school year. Fall data collection included contacting sampled schools to schedule appointments to conduct 
the child assessments in the spring of the school year, verify the parent consent procedures, link children 
to their teachers, identify children who had withdrawn from the school, and obtain locating information 
about their new schools. Spring data collection included the administration of direct child assessments 
and parent interviews and the collection of teacher and school questionnaires, student records abstracts, 
and facilities checklists. The activities to locate children and gain cooperation of the schools into which 
they transferred began in the fall and continued during the spring data collection. The content and 
timeline of the third grade data collection is shown in exhibit 4- 1 . 

Computer-assisted personal interviewing (CAPI) was the mode of data collection for the 
child assessments; telephone and in-person computer-assisted interviewing (CAI) was the mode of data 
collection for the parent interview; and self-administered questionnaires were used to gather information 
from teachers, school administrators, and student records. Field staff completed the facilities checklist. 



4.2 Roles and Responsibilities in the ECLS-K Study 

4.2.1 School’s Role 

During school recruitment, the schools were asked to designate a staff member to be the 
school coordinator to assist the ECLS-K staff with all school arrangements. 
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Exhibit 4-1. Timeline of third grade data collection: School year 2001-02 





Third grade 




2001 




2002 


Fall 


Winter 


Spring 


Advance school 


Advance school 


Child assessments 


contact 

1 


contact 

1 


conducted 

1 


1 

Tracing sampled 


1 

Tracing sampled 


1 

Parent interviews 


households 


households 


conducted 

1 

Teacher 

information 

collected 

1 

School and school 
administrator data, 
facilities checklist 
collected 

1 

Student records data 
collected 

1 

Tracing children 
who moved 

1 

Activities to gain transfer 
school cooperation 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Since the child assessments were administered at the schools, schools needed to provide 
appropriate space for conducting the assessments. 



4.2.2 School Coordinator’s Role 

A school coordinator was designated by the principal to facilitate the ECLS-K activities in 
the school. The school coordinator played a significant role in the smooth functioning and successful 
completion of the ECES-K child assessments in each cooperating school. He or she knew the personality 
of the school, the most opportune times to schedule the assessments, the available locations where the 
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one-on-one assessments could be conducted, and the best way to notify students, their parents, and their 
teachers of the assessment. 



The coordinator was asked to assist the ECLS-K in four ways: 

1 . Notify selected students, their teachers, and their parents of the study; 

2. Arrange for suitable space for the assessment activities; 

3. Provide information on sampled children, such as their grade and teacher’s name; and 

4. Distribute teacher and school questionnaires. 



4.2.3 Supervisor’s Role 

There were a total of 85 supervisors during the third grade data collection. Eight of the 
supervisors oversaw interviewers conducting the parent telephone interview. The remaining 77 oversaw 
field staff conducting both the parent interviews and child assessments. The supervisors’ responsibilities 
were as follows: 



■ Contact each school assigned to them to 

1 . Arrange for space to conduct the assessment; 

2. Schedule the spring assessment in original schools; 

3. Discuss parental contact information and consent procedures; 

4. Eink children to teachers; and 

5. Identify children who had withdrawn from the school; 

■ Follow up and track receipt of parental consent forms, as necessary; 

■ Update the Field Management System (FMS) regularly and report to their field 
manager; 

■ Transmit updated FMS data to the home office; 

■ Pick up e-mail regularly; and 

■ Return all materials at the end of the field period. 
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4.2.4 



Assessor’s Role 



A team of three to four assessors worked with each supervisor in a work area. The primary 
responsibilities of the ECLS-K assessors were to conduct the computer-assisted one-on-one child 
assessments and parent interviews. There were a total of 319 assessors, with 61 having conducted the 
parent telephone interview only (these 6 1 assessors are referred to as interviewers rather than assessors in 
the remainder of this chapter). The remaining 258 conducted both the parent interview and the child 
assessments. 



In addition to these responsibilities, some assessors were asked by their supervisor to assist 
with various other activities that took place in the school. These included, but were not limited to, 
assisting with sampling activities, preparing parental consent forms (if required), collecting teacher 
questionnaires, and assisting with various other recordkeeping tasks. 



4.2.5 Field Manager’s Role 

Eight experienced regional field managers were assigned to oversee the work of the 85 
supervisors. The field managers received regular reports from supervisors. They held weekly telephone 
conference calls with each supervisor assigned to them. If a supervisor had an immediate problem, he or 
she was encouraged to call the field manager at any time. 

Depending on the stage of the field period, the telephone conference calls between 
supervisors and field managers reviewed those activities that were in the planning stage, in progress, or in 
the process of being completed. These discussions included the following topics: 

■ Status of telephone contacts with original and transfer schools; 

■ Status of assessments scheduled in original schools by work area; 

■ Status of parent consent and followup in original and transfer schools; 

■ Status of linking children to teachers; 

■ Status of children who were withdrawn from the school; 
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■ Any refusal cases; 

■ Receipt of all school materials; and 

■ Overall and individual costs in the work area. 



4.3 Field Staff Training 

Several in-person training sessions were conducted to prepare staff for the third grade data 
collection. In the fall of 2001, supervisors were trained to contact original schools and recruit transfer 
schools. In the spring of 2002, four training sessions were held: one to train trainers, one for staff who 
only conducted parent interviews, one for field supervisors, and one for assessors. All training sessions 
were conducted using scripted training manuals to ensure that all trainees received the same information. 
Training sessions consisted of interactive lectures, scripted role plays, interactive exercises, and self- 
administered exercises. Interactive lectures were lectures with discussion and questions occurring 
periodically during the lecture. Scripted role plays usually consisted of pairs of trainees each pretending to 
be, for example, the assessor and the child or the interviewer and the parent. Such role plays gave trainees 
a chance to become more familiar with their materials and duties. Interactive exercises were group 
exercises led by the trainer in which all trainees participated. Self-administered exercises were, as the 
name suggests, completed by trainees working independently. Because of the complexity of the ECLS-K, 
trainees were required to become familiar with the functionality of their laptop computers and with the 
programs installed on them. Trainees were also required to become familiar with the different child 
assessment materials. See chapter 2 of the Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 (ECLS-K) Psychometric Report for the Third Grade (U.S. Department of Education, National 
Center for Education Statistics, forthcoming) for a description of the child assessment materials. The 
following sections discuss the fall and spring trainings. 



4.3.1 Advance Contact and Recruitment Training 

During the fall 2001, advance contact was made with the schools in order to remind them 
about the study and to collect information that would be helpful in the spring 2002 national data 
collection. The advance effort not only reduced the burden on supervisors in the spring but also 
reacquainted schools with the study’s procedures and gave supervisors a chance to encourage the schools’ 
participation. The major fall tasks were to contact schools to set appointments for the child assessments in 
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the spring, to verify the parent consent procedures, to link children to teachers, and to identify children 
who had withdrawn from the school and obtain locating information about their new schools. Field 
supervisors were trained for 3 days in September 2001 to contact original sampled schools and transfer 
schools to set up the data collection in the spring. A total of 50 field supervisors and 2 field managers 
completed training. Topics included an overview of study activities to date, verifying parent consent 
procedures, identifying and locating children who moved from the schools they attended in the first grade, 
identifying the teachers of ECLS-K children and linking them to those children, and exercises on 
scheduling schools efficiently within a work area (see appendix Al: Advance Contact Training Agenda). 

As in the first grade training, advance contact and recruitment training was conducted using 
the automated FMS. The FMS is a database that is used throughout the data collection period to enter 
information about the sampled children, parents, teachers, and school and to monitor production on all 
data collection activities. The FMS contains information essential to conducting and monitoring the 
progress of the data collection. For example, it contains the names and addresses of each school, the 
principal’s name and telephone number, the name and telephone number of the school coordinator, the 
first and last day of classes, the school hours, and the names of the sampled children in the school. For 
children, it contains information on their name and their parent’s name, whether they have any 
accommodations or need to use a hearing aid or glasses, as well as other information. Information of the 
same sort is also collected about the teachers and parents. For example, the FMS contains the name, 
address, and telephone number of each parent. For each parent, the name of the sampled child (or 
children) is also listed. For teachers, the FMS contains their name, school, and the sampled children in 
their classroom. It also indicates whether they are a regular or a special education teacher. To monitor 
production, the FMS contains case disposition codes and case assignment information. During training 
presentations, the field supervisors entered information into the FMS, thus acquiring hands-on experience 
with the FMS and all field procedures prior to beginning data collection. The field supervisors also 
completed role-play exercises that involved entering information into the FMS. 



4.3.2 Spring-Third Grade Training 

Field supervisors, interviewers, and assessors were trained for the spring- third grade data 
collection in three sessions in February and March 2002. The first session was trainers’ training and 
certification. During the second session, staff assigned to conduct only the parent interviews were trained. 
The last and largest training, held in March, involved the training of the supervisors and assessors. Before 
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the March in-person training session, supervisors and assessors completed 8 hours of home study training 
on the study design, field procedures, and computer keyboard skills. Staff conducting only the parent 
interviews did not have a pre-session home study to complete. 



4.3.2, 1 Trainers’ Training and Certification 

The purpose of trainers’ training was to (1) introduce lead and co-trainers to the training 
materials; (2) evaluate the flow, language, exercises, and time allotment of the training sessions; and 
(3) certify trainers on the child assessment. Some, but not all, co-trainers and runners (staff who assisted 
trainees who experienced difficulties with the CAI application and helped with the management and 
distribution of training materials) were also certified on the child assessment. Experienced trainers with 
in-depth knowledge of the ECLS-K conducted the trainer training sessions. Not only had these trainers 
developed the CAI specs, but they had also worked with expert consultants to develop the child 
assessments and the assessment materials. In addition, they conducted 9 assessments on nonsampled 
children between January and Febmary 2002 in order to become certified on the assessments of the 
ECES-K prior to the launch of the national data collection. 

Approximately 15 lead trainers, 15 co-trainers, and 15 runners were trained at trainers’ 
training in a single room. As noted earlier, experienced trainers conducted the parent interview and the 
child assessment training sessions. In addition, a data display person responsible for running the 
electronic data display and two runners assisted in the training. Trainers’ training was conducted for 5 
days in February 2002 in Rockville, Maryland. The trainers’ training agenda (see appendix A2: Trainer 
Training Agenda) covered many of the same topics that were presented during the assessment and parent 
interview training for the national data collection, as well as some additional topics on the trainers’ tasks 
and responsibilities. In addition, trainers were certified on the child assessment following the procedures 
described in section 4.3.2. 2. The only difference between trainer certification and field staff certification 
was that trainers were certified on an entire child assessment and the field staff were certified on 
approximately half of a child assessment. 

Parent Interviewer-Only Training. Supervisors and staff assigned to complete only parent 
interviews during the spring data collection attended a 2-day training in February 2002 in Rockville, 
Maryland. Eight supervisors and 6 1 interviewers completed training. Trainers presented the content of the 
parent interview and discussed protocols for interviewing (see appendix A3 : Parent Interviewer Training 
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Agenda). The interviewers practiced using the CAI system on laptop computers during interactive 
lectures and role plays (see section 4. 3.2. 3 for a discussion of Parent Interview Certification). Supervisors 
had an additional day of training in which they reviewed using the FMS to organize and track production 
and discussed management techniques for overseeing their teams of interviewers. 

Field Supervisor Training, Field supervisors had 3 days of training, 1 day preceding and 2 
days following the assessor training in Los Angeles, California. Field supervisors were also trained to use 
the FMS, and the field supervisors entered information into the FMS during training presentations. 
Seventy-seven field supervisors completed training. The topics covered in the field supervisor training 
session included reviewing materials from the fall school recruitment, role playing to practice contacting 
school coordinators, identifying and locating children who moved from their first grade schools, 
identifying the regular and special education teachers of ECLS-K children and linking them to those 
children, distributing and following up on teacher and school administrator questionnaires, completing the 
facilities checklist, and conducting quality control observations (see appendix A4: Supervisor Training 
Agenda). 



Assessor Training, The assessor training sessions were conducted in Los Angeles, 
California. Assessor training lasted for 5 days; field supervisors were also trained to perform all assessor 
activities. Two hundred sixty-six assessors^ and 77 field supervisors completed training. Assessor training 
included an overview of study activities to date, interactive lectures on the direct child assessments and 
the parent interview, role-play scripts to practice parent interviews and direct child assessments, direct 
child assessment precertification exercises on each form of the direct child domain assessments, 
techniques for parent refusal avoidance, and strategies for building rapport with children (see appendix 
A5: Assessor Training Agenda). A major goal of the assessor training was to train field staff in the proper 
procedures to conduct the direct child assessments. The sessions provided trainees with practical 
experience with all the direct child assessment materials and procedures and the CAI programs before 
data collection. Trainees practiced entering information into the CAI system on laptop computers during 
training sessions on conducting the direct child assessments and parent interview role-play scripts. 



'Eight of the 266 trainees chose to leave the study prior to the start of data collection, leaving 258 assessors working in the field. 
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4J.2.2 



Certification of the Child Assessors 



Assessors and field supervisors conducted child assessments. Training to administer the 
assessment battery included exercises that took into account the different ways children may answer 
questions. Ambiguous answers were included in both training interactives and role plays to provide 
practice in coding a variety of responses. Particular attention was paid to the question-by-question 
specifications (QxQs) for constructed response/open-ended questions so that a clear understanding of 
coding guidelines was established. These guidelines were reviewed frequently during training so that 
trainees were prepared for answers that were not phrased exactly as found in the scoring rubrics. 

Training on the specific subdomains was implemented in four parts. The first two parts 
involved precertification activities and the second two parts involved certification activities. The 
precertification activities involved an interactive lecture in which the trainer reviewed the specific items 
within the subdomain (i.e., reading, mathematics, and science test forms) using the QxQs and a 
standardized training script and individual practice, which addressed both item administration skills and 
accurate coding of responses. The certification activities involved written certification exercises and a live 
certification session during which each trainee conducted a portion of the assessment with an actual child 
respondent. 



The first part of the precertification training for each subdomain was an interactive lecture 
using the QxQs, which highlighted the constructed response/open-ended assessment items. Although 
conceivably children could produce a variety of answers for any item, there are only 47 questions 
(16 percent of the items) in the entire assessment that were open-ended items and would be expected to 
have the most variation. A constructed response/open-ended question is a short answer question that is 
scored against a specific criterion or a scoring rubric (scoring guide). These types of questions require the 
assessor to interpret and use multiple criteria in evaluating the responses. The assessor compared a child’s 
response to a scoring rubric in order to determine if the child’s response was correct or incorrect. Most of 
the open-ended assessment items were in the reading and science domains. 

The second part of the precertification training for each subdomain involved individual 
practice. Once an interactive lecture was completed, the trainees practiced administering the subdomain 
that had just been presented. The practice was not scripted. However, trainees were instructed to 
concentrate on practicing item administration skills, such as reading verbatim from the laptop computer 
screen and gesturing. Trainers observed trainees during the individual practice, recorded the results of the 
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individual practice in the Trainee Evaluation Form (see appendix Bl: Trainee Evaluation Form), and gave 
each trainee specific feedback on performance. The trainer reviewed the item administration skills with 
the trainee and provided additional instruction to trainees requiring additional practice. 

The third part of the training on each subdomain involved written certification exercises (see 
appendix B2: Certification Exercises). These exercises concentrated on the open-ended questions and 
were designed to evaluate the trainees’ ability to accurately select the appropriate response codes. Each 
exercise presented a constructed response/open-ended question with a variety of responses (some from 
the scoring rubric as well as some variations on those) and the assessor was required to code each 
response as either correct or incorrect. Training staff scored these exercises and feedback was given to the 
trainee on coding accuracy. Most trainees passed the written exercises on their first attempt. Just over a 
quarter of the trainees (24 percent or 84 trainees) did not pass at least one element of the certification 
exercises on their first attempt. The trainees who did not pass the written certification exercises completed 
specified remedial training steps prescribed by training staff. Once those remedial activities were 
completed, the trainee re-took the written exercise(s). This process was repeated until the trainee passed 
the exercise or the training staff determined that the trainee should be released. No field staff were 
released because of failure on the certification exercises. 

Training staff implemented a variety of remedial training steps for individual trainees and 
trainees collectively. The QxQs and interactive lectures were reviewed as necessary and the trainee(s) 
practiced the subdomain test again. The help laboratory was made available after the daily training 
session to provide additional instruction and help to trainees before the live portion of the child 
assessment certification. 

The culmination of the child assessments training was administering the cognitive 
assessment battery to children. This allowed the trainees to experience what it is like in the field and to 
put into practice all their new skills. Staff already certified on the child assessment observed trainees 
during the child assessment and gave feedback on performance using the Assessment Certification Form 
(see appendix B3: Assessment Certification Form). 

Supervisors and assessors were certified on the child assessment by administering 
approximately half of a cognitive assessment to a child while being observed by certified evaluators. As 
evaluators observed the assessment, they completed the appropriate sections of the Assessment 
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Certification Form. This form had both the trainee and the evaluator’s names recorded on the cover as 
well as the date the evaluation was conducted. 

The Assessment Certification Form had two sections: section 1 (exhibit 4-2) rated the trainee 
on key skill areas, such as building rapport, using neutral praise, responding to behaviors presented by the 
child, appropriate pacing, and avoiding coaching. The evaluator marked each skill area that the trainee did 
not demonstrate appropriately. 

Exhibit 4-2. Section 1 of the Assessment Certification Form: School year 2001-02 

Evaluator: As the assessment is administered, record whether or not the assessor successfully performed 
the following behaviors. Check “No” if the assessor makes 3 or 4 errors and needs to make improvements. 



SECTION 1 : Rapport building and working with the child 



Q: Did the assessor... 


No 


1 . Establish rapport with the child? 




2. Use an appropriate pace with the child? 




3. Use neutral praise? 




4. Respond to behaviors presented by child? 




5. Avoid coaching the child? 





SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Section 2 listed specific questions from each routing and subdomain form. The instructions 
for completing section 2 are shown in exhibit 4-3. For each of the listed questions observed, the evaluator 
recorded both the child’s response and noted if the trainee did not demonstrate the specified required 
administration skills for that question. The required administration skills included reading questions 
verbatim, using appropriate probes, and using appropriate hand motions (gesturing). For each question on 
which the evaluator observed that the trainee did not demonstrate the required administration skill(s), he 
or she checked a box, indicating which skill was not performed. 

As mentioned earlier, during the practice child assessment, the evaluator simultaneously 
coded the child’s response to the open-ended assessment items listed in the Assessment Certification 
Form. These open-ended items were flagged in the CAI program for quality control review and a screen 
showing how the trainee coded each of those answers to the questions was accessed at the end of the 
assessment for review with the evaluator as shown in exhibit 4-4. 
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Exhibit 4-3. Instructions for Section 2 of the Assessment Certification Form: School year 2001-02 



SECTION 2 : Specific Assessment Activities 

Supervisor/Evaluator: Code the items as the assessor administers the assessment. 

Code the child’s response as the item is administered. 

If the item requires probing, check the box if the assessor does not use the appropriate probe. 

Check the box in the “Verbatim” column if the assessor does not read the item exactly as worded on the 
screen. 

Check the box in the “Gesturing” column if the assessor does not use appropriate hand motions. 

For each validation code item, check the box in the “Validation” column if the response coded by the 
assessor is not what you have coded. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Exhibit 4-4. Quality review sereen: Sehool year 2001-02 




SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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Exhibit 4-5 presents an example of a question from the Assessment Certification Form, which 
required the evaluator to evaluate the required assessment administration skills: reading verbatim, using 
appropriate gesturing, using the appropriate probe, and accurately coding the child’s response to this open- 
ended question. The evaluator checked the boxes only if the assessor did not demonstrate the correct 
behavior. For all questions observed by the evaluator and listed in the Assessment Certification Form, the 
evaluator recorded if the assessor did not read verbatim or gesture appropriately. Evaluators only marked a 
box when the assessor did not use the appropriate probe for those questions that displayed probes (a box 
labeled “Assessor Used Inappropriate Probe?” in the Assessment Certification Form). Also, the evaluators 
only marked the box if their response code did not match the assessors’ for open-ended questions (a box 
labeled “Validation Code Did Not Match” was presented in the Assessment Certification Form). 



Exhibit 4-5. Sample question from Assessment Certification Form: School year 2001-02 





DID NOT 
READ 
OUESTION 
VERBATIM 


GESTURED 

INAPPRO- 

PRIATELY 


VALIDA-TION 
CODE DID 
NOT MATCH 


R3.285 What do you think is the difference between the 
storyteiler’s bedroom and the sister’s bedroom?** 

IF STUDENT SAYS, “ A mess” OR “A lot of junk”, 
PROMPT: “Flow is it different?” 

CORRECT 1 

STORY TELLER’S BEDROOM IS: 

(BIGGER MESS/MORE 
MESS/MESSIER) 

(MORE JUNK/JUNKIER) 

(IS A MORE DISGUSTING SPOT) 

CORRECT ANSWER MUST INVOLVE 
A COMPARISON 

INCORRECT 2 

(STORYTELLER’S BEDROOM IS 

CLEAN/AS MESSY AS SISTER’S BEDROOM) 
DETAILS ABOUT THE SISTER’S 
BEDROOM 
EXAMPLES: 

(PIZZA UNDER THE DESK) 

(SOCKS ON THE LIGHT) 

(MESS/ALSO A MESS) 

(JUNK/ A LOT OF JUNK) 

REFUSED 777 

DON’T KNOW 999 

ASSESSOR USED INAPPROPRIATE PROBE? □ 


□ 


□ 


□ 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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At the end of the “live” child assessment, after the child had been escorted from the room, 
the evaluator and the trainee reviewed each trainee’s overall performance on the half of the cognitive 
assessment that he or she conducted. After discussing the ratings in section I, the trainee accessed the 
quality control screen. The trainee and evaluator reviewed their codes for each open-ended question asked 
in section II of the Assessment Certification Form. 

The evaluator then scored the certification assessment using the scoring form shown in 
exhibit 4-6. The evaluator first counted the number of boxes checked in section I: Rapport Building of the 
Assessment Certification Form (see exhibit 4-2), and recorded that number in the appropriate row of 
Form A of the Certification Scoring Form (exhibit 4-6). The evaluator then counted the number of check 
marks for each section of the assessment that was observed and recorded those numbers in the appropriate 
boxes of Form A. Exhibit 4-7 presents an example of a completed Form A. In this example, the evaluator 
observed one trainee who administered Reading Routing, Reading Yellow and Math Routing. The 
evaluator recorded zero (0) check marks for section I, zero (0) check marks for Reading Routing, one (1) 
check mark for Reading Yellow, and zero (0) check marks for Math Routing. The total check marks 
recorded was one (1). 

The evaluator then calculated the number of possible points for the sections of the 
assessment observed using the Total Possible Points Chart displayed in exhibit 4-8. This chart was 
necessary for the assessment certification because each trainee only conducted half of the cognitive 
assessment with a child respondent; this chart helped the evaluator use the correct denominator for 
calculating the score. For each section of the assessment observed, the evaluator circled the last question 
recorded in the Assessment Certification Form and wrote the number of possible points in the appropriate 
box in the second section of the scoring form (Form B) (see exhibit 4-9). Continuing with the example 
started earlier, the evaluator observed all of Reading Routing (6 possible points), all of Reading Y ellow 
(24 possible points), and all of Math Routing (7 possible points). In addition, section 1 contributes 5 
possible points. Based on these sections of the assessment, the total number of possible points for this 
trainee is 42. 
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Exhibit 4-6. Certification Scoring Form: School year 2001-02 



Certification Scoring 

Record the number of check marks from section 1: Rapport Building and section 2: Specific 
Assessment Activities in the appropriate boxes of Form A. 

Sum each row and record total in Row Totals column of Form A. 

Sum Row Totals column (including number of check marks from section 1) and record in Total 
check marks box of Form A. 



Form A: Checks Marks Recorded in Sections 1 and 2 



Assessment 


Routing Form 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 












Mathematies 












Seienee 












Number of eheek marks from section 1: Rapport Building: 




Total Cheek Marks: 





Using the Total Possible Points Chart, circle the last question in the assessment certification 
form for each routing and subdomain that the trainee completed. Put the number to the right of 
the item in the appropriate box of Form B. For example, the Trainee completed Reading 
Routing and through item R3.185 of Reading Red, the total number of possible points is 19 (6 
for Routing and 13 for Red). Record 6 in the Reading row under Routing Form and record 13 
under Red Form and 19 in the Row Totals column of Form B. 

Sum Row Totals column (including Possible Points for section 1) and record in Total Possible 
Points box of Form B. 



Form B: Total Possible Points 



Assessment 


Routing Form 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 












Mathematies 












Seienee 












Possible Points fox section 1: Rapport Building: 


5 


Total Possible Points: 





Step 6: Use Proportion Correct Chart to determine the proportion correct and write that proportion in 

this box: | | 



Result: If Proportion correct is greater than or equal to .85, Assessor is Certified. 

If Proportion correct is between .70 and .85, Assessor requires Remedial Action (Complete 
Remedial Task Chart below) 

If Proportion correct is less than .70, Assessor Failed. 



Step 4: 



Step 5: 



Step 1: 

Step 2: 
Step 3: 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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Exhibit 4-6. Certification Scoring Form: School year 2001-02 — Continued 



Remedial Task Chart 

Circle the number of the task(s) needing improvement. Discuss remedial action with the assessor. 


TASK 


# 


REMEDIAL ACTION 


APPROPRIATE PACE 


I 


PRACTICE/REVIEW JOB AIDS 


AVOID COACHING 


2 


REVIEW THE JOB AIDS 


READ VERBATIM 


3 


PRACTICE READING FROM THE SCREEN 


GESTURING APPROPRIATELY 


4 


PRACTICE GESTURING 


USE APPROPRIATE PROBES 


5 


REVIEW THE JOB AIDS/Q-BY-Qs 


VALIDATION CODE DID NOT MATCH 


6 


REVIEW THE Q-BY-Qs 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Exhibit 4-7. Example of Form A for Trainee 1 : School year 2001-02 



Form A: Checks Marks Recorded in Sections 1 and 2 



Assessment 


Routing Form 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 


0 




1 




1 


Mathematics 


0 








0 


Science 












Number of check marks from section 1: Rapport Building: 


0 


Total Check Marks: 


1 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 
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Exhibit 4-8. Total Possible Points Chart: School year 2001-02 



Certification Scoring 
Total Possible Points Chart 



Reading 



Routing 


Red 


Yellow 


Blue 


Item # 


Total 

Points 


Item # 


Total 

Points 


Item # 


Total 

Points 


Item # 


Total 

Points 


R3.002 


3 


R3.no 


3 


R3.230 


2 


R3.420 


2 


R3.065 


6 


R3.155 


6 


R3.255 


5 


R3.425 


5 




R3.175 


10 


R3.275 


9 


R3.455 


8 


R3.185 


14 


R3.285 


13 


R3.535 


10 


R3.190 


17 


R3.300 


16 


R3.540 


13 


R3.200 


21 


R3.325 


18 


R3.550 


16 






R3.330 


21 


R3.585 


19 




R3.360 


24 


R3.595 


22 



Mathematics 



Routing 


Red 


Yellow 


Blue 


Item # 


Total 


Item # 


Total 


Item # 


Total 


Item # 


Total 




Points 




Points 




Points 




Points 


M3.035 


2 


M3. 135 


3 


M3.230 


2 


M3. 345 


2 


M3.060 


5 


M3. 140 


5 


M3.250 


5 


M3.385 


5 


M3.075 


7 


M3. 155 


8 


M3.260 


7 


M3.390 


8 






M3. 175 


10 


M3.290 


10 


M3. 420 


11 






M3.205 


12 


M3.300 


14 


M3.435 


14 










M3.305 


16 


M3. 445 


17 














M3.450 


20 



Science 



Routing 


Red 


Yellow 


Blue 


Item # 


Total 

Points 


Item # 


Total 

Points 


Item # 


Total 

Points 


Item # 


Total 

Points 


S3.015 


3 


S3.095 


2 


S3. 185 


2 


S3.285 


3 


S3.020 


6 


S3. 155 


4 


S3.210 


5 


S3.286 


6 


S3.040 


9 


S3. 170 


7 


S3.230 


7 


S3.305 


9 




S3. 175 


9 


S3.245 


9 


S3.315 


13 






S3.246 


13 


S3.320 


17 




S3.260 


17 


S3.365 


19 


S3.265 


21 





SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 
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Exhibit 4-9. Example of Form B: School year 2001-02 



Form B: Total Possible Points 



Assessment 


Routing Form 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 


6 




24 




30 


Mathematics 


7 








7 


Science 












Possible Points for section 1: Rapport Building: 


5 


Total Possible Points: 


42 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 



Finally, the evaluator determined the proportion correct by using the Proportion Correct 
Chart displayed in exhibit 4-10. Continuing the example, the evaluator found the column on the 
Proportion Correct Chart that displayed the total possible points for the trainee based on the portions of 
the assessment observed (42) and the row on the Proportion Correct Chart that displayed the total check 
marks recorded for the trainee (1), and recorded the proportion from the corresponding box on the 
Proportion Correct Chart in the box under Step 6 (exhibit 4-11) on the scoring form (.98). Once the 
evaluator completed the scoring, the trainee was rated as Certified, Remedial Action, or Failed. 
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Number of Check Marks 



Exhibit 4-10. Proportion correct chart: School year 2001-02 

Certification Scoring 
Proportion Correct Chart 



Total Possible Points 





19 


20 


21 


22 


23 


24 


25 


26 


27 


28 


29 


30 


31 


32 


33 


34 


35 


36 


37 


38 


39 


40 


1 


0.95 


0.95 


0.95 


0.95 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.98 


2 


0.89 


0.90 


0.90 


0.91 


0.91 


0.92 


0.92 


0.92 


0.93 


0.93 


0.93 


0.93 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.95 


0.95 


0.95 


0.95 


3 


0.84 


0.85 


0.86 


0.86 
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0.88 
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0.88 


0.89 


0.89 


0.90 


0.90 


0.90 


0.91 


0.91 


0.91 


0.91 


0.92 


0.92 


0.92 


0.92 


0.93 
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0.79 


0.80 


0.81 


0.82 


0.83 


0.83 


0.84 


0.85 


0.85 


0.86 


0.86 


0.87 


0.87 


0.88 


0.88 


0.88 


0.89 


0.89 


0.89 


0.89 


0.90 


0.90 
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0.74 


0.75 


0.76 


0.77 


0.78 


0.79 


0.80 


0.81 


0.81 


0.82 


0.83 


0.83 


0.84 


0.84 


0.85 


0.85 


0.86 


0.86 


0.86 


0.87 


0.87 


0.88 
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0.68 


0.70 


0.71 


0.73 


0.74 


0.75 


0.76 


0.77 


0.78 


0.79 


0.79 


0.80 


0.81 


0.81 


0.82 


0.82 


0.83 


0.83 


0.84 


0.84 


0.85 


0.85 


7 


0.63 


0.65 


0.67 


0.68 


0.70 


0.71 


0.72 


0.73 


0.74 


0.75 


0.76 


0.77 


0.77 


0.78 


0.79 


0.79 


0.80 


0.81 


0.81 


0.82 


0.82 


0.83 
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0.58 


0.60 


0.62 


0.64 


0.65 


0.67 


0.68 


0.69 


0.70 


0.71 


0.72 


0.73 


0.74 


0.75 


0.76 


0.76 


0.77 


0.78 


0.78 


0.79 


0.79 


0.80 


9 


0.53 


0.55 


0.57 


0.59 


0.61 


0.63 


0.64 


0.65 


0.67 


0.68 


0.69 


0.70 


0.71 


0.72 


0.73 


0.74 


0.74 


0.75 


0.76 


0.76 


0.77 


0.78 
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0.47 


0.50 


0.52 


0.55 


0.57 


0.58 


0.60 


0.62 


0.63 


0.64 


0.66 


0.67 


0.68 


0.69 


0.70 


0.71 


0.71 


0.72 


0.73 


0.74 


0.74 


0.75 
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0.42 


0.45 


0.48 


0.50 


0.52 


0.54 


0.56 


0.58 


0.59 


0.61 


0.62 


0.63 


0.65 


0.66 


0.67 


0.68 


0.69 


0.69 


0.70 


0.71 


0.72 


0.73 
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0.37 


0.40 


0.43 


0.45 


0.48 


0.50 


0.52 


0.54 


0.56 


0.57 


0.59 


0.60 


0.61 


0.63 


0.64 


0.65 


0.66 


0.67 


0.68 


0.68 


0.69 


0.70 
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0.32 


0.35 


0.38 


0.41 


0.43 


0.46 


0.48 


0.50 


0.52 


0.54 


0.55 


0.57 


0.58 


0.59 


0.61 


0.62 


0.63 


0.64 


0.65 


0.66 


0.67 


0.68 
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0.26 


0.30 


0.33 


0.36 


0.39 


0.42 


0.44 


0.46 


0.48 


0.50 


0.52 


0.53 


0.55 


0.56 


0.58 


0.59 


0.60 


0.61 


0.62 


0.63 


0.64 


0.68 
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0.21 


0.25 


0.29 


0.32 


0.35 


0.38 


0.40 


0.42 


0.44 


0.46 


0.48 


0.50 


0.52 


0.53 


0.55 


0.56 


0.57 


0.58 


0.59 


0.61 


0.62 


0.63 


















































41 


42 


43 


44 


45 


46 


47 


48 


49 


50 


51 


52 


53 


54 


55 


56 


57 


58 


59 


60 


61 


62 


1 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 
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0.95 


0.95 


0.95 


0.95 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.97 


0.97 


0.97 


0.97 


0.97 
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0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 
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0.90 


0.90 


0.91 


0.91 


0.91 


0.91 


0.91 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.94 
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0.88 


0.88 


0.88 


0.89 


0.89 


0.89 


0.89 


0.90 


0.90 


0.90 


0.90 


0.90 


0.91 


0.91 


0.91 


0.91 


0.91 


0.91 


0.92 


0.92 


0.92 


0.92 
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0.85 


0.86 


0.86 


0.86 


0.87 


0.87 


0.87 


0.88 


0.88 


0.88 


0.88 


0.88 


0.89 


0.89 


0.89 


0.89 


0.89 


0.90 


0.90 


0.90 


0.90 


0.90 
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0.83 


0.83 


0.84 


0.84 


0.84 


0.85 


0.85 


0.85 


0.86 


0.86 


0.86 


0.87 


0.87 


0.87 


0.87 


0.88 


0.88 


0.88 


0.88 


0.88 


0.89 


0.89 
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0.80 


0.81 


0.81 


0.82 


0.82 


0.83 


0.83 


0.83 


0.84 


0.84 


0.84 


0.85 


0.85 


0.85 


0.85 


0.86 


0.86 


0.86 


0.86 


0.87 


0.87 


0.87 
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0.78 


0.79 


0.79 


0.80 


0.80 


0.80 


0.81 


0.81 


0.82 


0.82 


0.82 


0.83 


0.83 


0.83 


0.84 


0.84 


0.84 


0.84 


0.85 


0.85 


0.85 


0.85 
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0.76 


0.76 


0.77 


0.77 


0.78 


0.78 


0.79 


0.79 


0.80 


0.80 


0.80 


0.81 


0.81 


0.81 


0.82 


0.82 


0.82 


0.83 


0.83 


0.83 


0.84 


0.84 
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0.73 


0.74 


0.74 


0.75 


0.76 


0.76 


0.77 


0.77 


0.78 


0.78 


0.78 


0.79 


0.79 


0.80 


0.80 


0.80 


0.81 


0.81 


0.81 


0.82 


0.82 


0.82 
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0.71 


0.71 


0.72 


0.73 


0.73 


0.74 


0.74 


0.75 


0.76 


0.76 


0.76 


0.77 


0.77 


0.78 


0.78 


0.79 


0.79 


0.79 


0.80 


0.80 


0.80 


0.81 
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0.68 


0.69 


0.70 


0.70 


0.71 


0.72 


0.72 


0.73 


0.73 


0.74 


0.75 


0.75 


0.75 


0.76 


0.76 


0.77 


0.77 


0.78 


0.78 


0.78 


0.79 


0.79 
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0.68 


0.69 


0.70 


0.70 


0.71 


0.72 


0.72 


0.73 


0.73 


0.74 


0.75 


0.75 


0.75 


0.76 


0.76 


0.77 


0.77 


0.78 


0.78 


0.78 


0.79 


0.79 
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0.63 


0.64 


0.65 


0.66 


0.67 


0.67 


0.68 


0.69 


0.69 


0.70 


0.71 


0.71 


0.72 


0.72 


0.73 


0.73 


0.74 


0.74 


0.75 


0.75 


0.75 


0.76 



NOTE: Proportion correct = (Total Possible Points - Number of Check Marks)/Total Possible Points. EXAMPLE: If an assessor received 3 check marks out of a total of 24 possible points, the 
proportion correct is .88. This Assessor would be certified. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, school year 
2001 - 02 . 





Exhibit 4-11. Example of Proportion Correct Chart: School year 2001-02 



Step 6: Use Proportion Correct Chart to determine the proportion correct and write that proportion in 

this box: I IT I 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Table 4-1 presents the results of the training certification. Trainees scoring 85 percent or 
above were certified as qualified to administer the child assessments. Trainees who scored between 70 
and 84 percent were required to complete remedial training and have an additional certification conducted 
before beginning assessments. 



Table 4- 1 . Number and percent of trainees, by scores on certification form: 
School year 2001-02 



Trainees' 


Number 


Percent 


Total 


343 


100 


Score on certification form 


85 percent or above 


337 


98.2 


70-84 percent 


6 


1.8 


Below 70 percent 


0 


0 



' The trainees consisted of 258 assessors and 77 field supervisors. Eight trainees chose to leave the 
study prior to the start of data collection, leaving 258 assessors working in the field. 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood 
Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 



The majority of the trainees (98.2 percent) scored above 85 percent on the certification form, 
with only 1.8 percent (n=6) who scored between 70 and 84 percent. None of the trainees failed to meet 
the 70 percent threshold on the Assessment Certification Form. All trainees who needed remedial training 
were certified qualified to administer the child assessments after they conducted a second assessment on a 
third grade-aged child who was not part of the ECES-K sample. 

The six trainees who required remedial action were assigned remedial training, exercises, 
and practice assessments to perform on an age-appropriate child in their area. Their supervisor observed 
the practice assessment and certified them. The scoring form specifies the type of remedial training 
activities that were assigned based on the type of problems observed by the evaluator (see the Remedial 
Task Chart, exhibit 4-12; appendix B4). Once the remedial training activities were completed, the field 
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supervisor or field manager was responsible for conducting an additional certification assessment with the 
trainee with a third grade-aged child not in the ECLS-K sample prior to being able to start work. 



Exhibit 4-12. Example of Remedial Task Chart: School year 2001-02 



Remedial Task Chart 

Circle the number of the task(s) needing improvement. Discuss remedial action with the assessor. 


TASK 


# 


REMEDIAL ACTION 


APPROPRIATE PACE 


1 


PRACTICE/REVIEW JOB AIDS 


AVOID COACHING 


2 


REVIEW THE JOB AIDS 


READ VERBATIM 


3 


PRACTICE READING FROM THE SCREEN 


GESTURING APPROPRIATEEY 


4 


PRACTICE GESTURING 


USE APPROPRIATE PROBES 


5 


REVIEW THE JOB AIDS/O-BY-Os 


VAEIDATION CODE DID NOT MATCH 


6 


REVIEW THE Q-BY-Qs 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



4.3.2,3 Parent Interview Training and Certification 

Training assessors to conduct the parent interview also included interactive lectures and role 
plays that were designed not only to review the intent of the questions but also to demonstrate the 
different ways that parents may answer questions. Ambiguous answers were included in both training 
interactive lectures and role-play scripts to provide practice in probing and handling a variety of 
responses. Parent interview QxQs were carefully reviewed throughout training so that 
assessors/interviewers would be prepared for a variety of responses from respondents. The culmination of 
training on the parent interview was a final certification role play that was conducted with the 
assessor/interviewer’s supervisor. This certification role play was designed to test all the protocols and 
techniques that were reviewed during training. 

After training, interviewers and assessors who would be conducting parent interviews 
completed a final role play with their field supervisor (see appendix B5: Parent Interview Certification 
Role Play). These role plays occurred in the first week after training before any interviews were done. 
The field supervisor conducted the final role play using a scripted parent interview and an evaluation form 
(see appendix B6: Parent Interview Certification Role Play Evaluation Form). The final role play script 
was designed to test the interviewer/assessor’s understanding of the interview content, proper 
interviewing techniques, including probing, and accurate response recording in CATE As they conducted 
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the interview, field supervisors completed the Parent Interview Final Role Play Evaluation Form. The 
evaluation form consisted of a checklist that supervisors used to evaluate interviewers/assessors in key 
areas, such as contacting and selecting the respondent, asking the questions verbatim, probing properly, 
and following the correct question path. Two points were given for each item performed correctly. 
Trainees had to score at least 30 out of a possible 38 points to pass. 

All of the interviewers (61) and the majority of assessors (233) conducted parent interviews. 
The majority of interviewers/assessors (94.8 percent or 244 staff) were successfully certified on the parent 
interview. Nine of the 244 staff certified on the parent interview were bilingual staff who were certified in 
Spanish. Supervisors also conducted some parent interviews. All 77 supervisors passed the certification. 
The majority of the final role plays were conducted in English (217 interviewers/assessors). 



4.4 Fall Preassessment School Contact 

Beginning in September 2001, all participating ECES-K schools (i.e., schools that 
participated in fall or spring of kindergarten or first grade) were contacted by telephone to prepare for the 
spring data collection. When children were identified as transferring to another school, the child’s new 
school (and district, if necessary) was recruited. As noted in section 4.3.1, the advance contact 
served several purposes. It reminded schools about the study and reacquainted them with the study’s 
procedures; it provided supervisors the opportunity to persuade the schools to participate; and it allowed 
the collection of information necessary for the spring 2002 national data collection. There were four 
primary tasks to be accomplished during the fall contact. These were to schedule appointments to 
conduct the child assessments in the spring; to verify parent consent procedures; to identify the children’s 
teachers; and to identify children who had withdrawn from the school and obtain locating information 
about their new schools. The fall contact activities are described below. 



4.4,1 Advance Mailings 

In September 2001, an advance package was mailed via Federal Express to all participating 
ECES-K schools asking them to prepare for the preassessment contact telephone call. The schools were 
asked to identify a school staff coordinator to serve as a liaison with the study (in original sampled 
schools, this person was usually the coordinator from the previous rounds of data collection). A package 



4-22 




containing study materials was sent to the schools. The package contained the following materials (see 
appendix C: Fall Preassessment Advance Contact Materials): 



■ Appendix Cl: Advance Letter (original schools) — a letter printed on ECLS-K 
letterhead reminding school staff about the study, describing the third grade data 
collection, and alerting the school coordinator of the advance contact in the fall; 

■ Appendix C2: Advance letter (transfer schools) — a letter printed on ECLS-K 
letterhead introducing school staff to the study, describing the third grade data 
collection, and alerting the school coordinator of the advance contact in the fall; 

■ Appendix C3: School Summary Sheet (original schools) — a two-page document 
providing a brief review of the study to date and the third grade data collection 
activities; 

■ Appendix C4: School Summary Sheet (transfer schools) — a two-page document 
providing a brief overview of the study and the third grade data collection activities; 

■ Appendix C5: Study Findings Sheet — a summary of findings about children from the 
previous rounds of data collection; 

■ Appendix C6: ECLS-K Study Children Form and Instructions — a listing of all the 
sampled children and instructions for completing the form with specific information 
such as the children’s continued attendance at the school, their grade, their teachers’ 
names, classroom numbers, receipt of special education services, and receipt of 
assessment accommodations/exclusions before the preassessment call, and: 

■ Appendix C7: Class Organization Form — a form for requesting information on 
departmentalized instruction in the schools 



4.4.2 Fall Preassessment School Coordinator Contact 

The preassessment contacts were made by telephone between September and November 
2001. The preassessment school contacts were successful in meeting all four tasks described above. 
Contacting original sampled schools to set up the spring assessment and identifying children who 
withdrew from their spring- first grade school and moved into their third grade transfer school, enabled the 
identification of schools that were ineligible for third grade data collection. Schools were determined to 
be ineligible for third grade data collection if no ECLS-K sampled children were currently enrolled. 
Original sampled schools became ineligible because second grade was the highest grade in the school or 
because the school had closed, that is, was no longer operational. More transfer schools were determined 
to be ineligible as children transferred out of them into other schools. 
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During the preassessment contact, the field supervisor contacted the school coordinator to 
collect some basic information about the school and some detailed information about each ECLS-K 
sampled child. The field supervisor used the School Information Form (see appendix D1 for the School 
Information form) to collect basic information about the school, such as school start and end dates, 
vacation and holiday schedules, and parking directions. The form was also used to determine if the school 
was a year-round school, taught third grade, or required new parent consent, and to obtain information on 
class organization. The supervisor used the Child Work Grid (see appendix D5 for the supervisor version 
of this form) to collect basic information about the child such as his or her grade, the name and classroom 
number of the child’s primary teacher to link the child to a teacher, and whether the child had an 
Individualized Education Plan (lEP) or its equivalent. In addition, in original sample schools, the 
assessment date was scheduled; assessment dates for new transfer schools were scheduled in the spring. 



4.4.2. 1 Reviewing Parent Consent With the School 

Although parental consent was obtained in the base year (and, in some schools, in the first 
grade year), field supervisors asked the school coordinator whether the base year or first grade parental 
consent was acceptable for third grade. If the schools required current consent form s or changed the type 
of consent that was required (e.g., from implicit to explicit), parent letters and consent forms were either 
mailed to the school for distribution to parents or sent directly to parents by Westat depending on the 
schools’ preference (see appendix D2: Consent Process Flowchart, which describes this process). Parent 
cover letters and consent forms were available in English and Spanish (see appendix D3: Parent Cover 
Fetters and Consent Forms). Overall, 16 percent of all schools contacted required an updated parental 
consent. Of the schools requiring updated parental consent, approximately 50 percent required explicit 
consent from the parents. 



4.4.2.2 Class Organization in the School 

As children move into the later elementary grades, they may have different teachers for 
different subjects. This practice is usually referred to as departmentalized instruction. For the ECFS-K 
data collection, knowing how classrooms are organized is important so that the children’s correct teacher 
for a given subject can be identified. In an effort to more fully understand the teaching approaches used in 
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schools and to begin the process of crafting an approach for the fifth grade data collection, the school 
coordinator was asked to complete the Class Organization Form (see appendix D4: Class Organization 
Form [Supervisor Version]). To minimize disruption to the school during spring data collection, details 
about how the school organized instruction for language arts, mathematics, science, and social studies 
were collected by the school coordinator prior to the preassessment call. The field supervisor then 
collected the class organization information during the preassessment call with the school coordinator. In 
schools that indicated they had some type of departmentalized instruction for third grade, as opposed to 
self-contained classrooms, additional information was collected about the subjects that the teachers of the 
sampled ECLS-K children taught. Table 4-2 presents the percentages of different instructional practices 
by grade for the 2,422 ECLS-K schools that reported about instructional practices. There appeared to be a 
tendency to increase the use of departmentalized instruction in the later grades with only 4.4 percent of 
schools reporting departmentalized instruction for third grade but 1 1.3 percent by fifth grade; only 39 of 
the original sample schools identified departmentalized instruction in their third grade classes. 



Table 4-2. ECLS-K schools, categories of classroom instruction by grade: School year 2001-02 



Which category best describes the way that the third, fourth, and 
fifth grade classes are organized at this school? 


Third 
grade, by 
percent 


Fourth 
grade, by 
percent 


Fifth 
grade, by 
percent 


Self-contained class — multiple subjects are taught to the same 
class of students all or most of the day 


82.9 


70.3 


59.9 


Team teaching — one or more teachers collaborate in teaching 
multiple subjects to the same class of students 


6.5 


9.4 


11.3 


Departmentalized instruction — subject matter courses, language 
arts, mathematics, science, social studies, are taught to several 
classes of different students all or most of the day 


4.4 


8.6 


13.8 


Unknown 


6.2 


11.0 


15.0 



NOTE: Data from preassessment call. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



4.4.2.3 Collecting Information About ECLS-K Sampled Children 

Field supervisors primarily used the Child Work Grid (see appendix D5 for the supervisor 
version of this form) to collect information from the school coordinator about the ECLS-K sampled 
children. The school coordinator was asked to complete the school version of the form to expedite the 
call. Field supervisors collected information from the school coordinators about the ECLS-K sampled 
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children still enrolled in the school, including the child’s current grade, the name and classroom for the 
child’s regular teacher in order to link the child to a teacher, and whether or not the child had an lEP (see 
appendix D6: Criteria for Assessing Children With Disabilities and Accommodations in the ECLS-K). If 
the child had an lEP, then the name and classroom of the child’s special education teacher was noted, 
along with whether the child required any accommodations to participate in the direct cognitive 
assessment. Through this process, 1,177 children were identified with lEPs and 891 special education 
teachers were linked to these children. The accommodations to the third grade direct cognitive assessment 
were the same as those for the kindergarten and first grade direct cognitive assessment (see section 4.5.2). 
Field supervisors contacted the teachers of the ECES-K children as necessary for any of this information. 
Field supervisors also identified the respondent for the student records abstract for each child (see 
appendix D7: Student Records Abstract Finking Form). 

If a child was identified as having transferred out of the school, the field supervisor asked the 
school coordinator to provide the names, addresses, and telephone numbers of these transfer schools. Of 
those children who transferred, only a subset were followed to their new school (see section 3.4.2 in 
chapter 3 for more detail on how mover children were subsampled). If the new school belonged to a 
district that was new to the study, the district was contacted and recruited before any contact was made 
with the school (see appendix D8: New District Recruitment Fetter). If the district was already 
cooperating, the district was notified and the new school was contacted and recruited directly (see 
appendix D9: Cooperating District Notification Fetter). Field supervisors also verified with the school 
that no child who had previously transferred had returned to the school (see appendix DIO: Returning 
Student Form). 



4.4.2.4 Contacting Families of Homeschooled Children 

As part of the fall preassessment contact, children in the ECFS-K sample who were 
homeschooled in previous rounds were identified. The status of home-schooled children who were 
identified in rounds 1 through 4 was verified with their parents and updated as necessary. In addition, 
during the preassesment contact some schools identified homeschooled children. Their status was also 
verified with their parents during data collection. Parents of these children were contacted by telephone in 
September through November 2001 to determine if the child was still homeschooled or had enrolled in a 
school (see appendix Dll: Homeschooled Children Form). If the child had enrolled in a school, the new 
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school was contacted and recruited into the study. Parents of children who were still schooled at home 
were notified about the next round of data collection in the spring. 



4.4.3 Fall Preassessment Contact Results 

The goals for the fall preassessment contact with schools were the following: (1) set 
appointments for the spring assessment in original sample schools, (2) identify schools that ended at 
second grade to determine the school to which the sample children transferred, (3) identify children who 
changed schools since first grade, (4) link children to teachers for the advance school and teacher 
questionnaire mailings, and (5) contact as many transfer schools as possible within the field period to 
ascertain whether the child was still there and recruit the school into the study. It was not expected that 
every transfer school identified within the fall contact could be contacted within the fall field period 
because of the numbers of children that were expected to move. It was also expected that additional 
schools would be contacted during the spring round because children were expected to continue to move 
between fall and spring of the school year. 

Approximately 30 percent of the ECLS-K sampled children transferred to other schools 
between the spring of first grade and beginning of third grade (i.e., fall 2001). At the start of the fall 2001 
field period, there were 1,410 transfer schools, but 506 were deemed out-of-scope resulting in 904 
transfer schools identified prior to the fall contact. An additional 858 transfer schools were identified 
during the fall contact field period (September-November) for a total of 1,762 transfer schools at the end 
of the field period. All of the transfer schools identified during the fall were contacted by mail in the 
spring with an advance package that included the name of the child who transferred into the school. In 
addition, attempts were made to follow up with the schools via telephone. However, telephone contact 
could be made with only 52 percent of the schools before the fall field period ended. During the fall 
preassessment contact, 4,679 children were identified as movers and processed in preparation for the 
spring data collection. Table 4-3 presents the status of these 4,679 mover cases. 
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Table 4-3. Status of movers identified during fall preassessment contact: School year 2001-02 



Status 


Total 


Percent 


Total 


4,679 


100 


Fielded for assessment in spring 


1,981 


42 


Assess in school 


1,804 


39 


Assess in home 


177 


4 


Unlocatable 


781 


17 


Moved to nonsampled PSU 


220 


5 


Moved outside of U.S. 


45 


1 


Subsample not followed 


1,145 


24 


End of field period 


507 


11 



NOTE: Movers who had a status of “Fielded for assessment in spring” were treated like the nonmover cases during the spring data collection. 
Those with a status of “Unlocatable” could not be located during the fall preassessment contact. Attempts to locate them continued into January 
and February 2001. The cases were fielded in the spring if the location efforts were successful (see section 4.4.4) Cases identified as “Moved to 
nonsampled PSU” or “Moved to outside of U.S.” were not fielded for the spring data collection because they had moved out of the designated 
data collection area. Cases with a status of “Subsample not followed” were not fielded in the spring because the sampling plan called for 
collecting data from only a subsample of movers (see chapter 3, section 3.4.2). These cases were not in the subsample to be followed. The 507 
cases with a status of “End of field period” were not contacted in the fall because the schools were either new to the school sample frame with 
limited information available to contact them or in school districts new to the sample that required an additional contact at the district level before 
the schools could be contacted for spring data collection. These cases were closed out and rolled over for contact in the spring. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



The fall preassessment contact protocol was completed for 98 percent of the original 
ECLS-K schools and 76 percent of the transfer schools (both those identified in the fall field period and 
those identified during the first grade data collection) within the fall field period. Seventeen percent of 
schools were identified as out of scope, since they did not contain any sampled students (6 percent of 
original sampled schools and 22 percent of transfer schools). All of the schools that children transferred to 
as a result of the school ending at second grade, closing, or merging with another school were identified 
within the field period. Tables 4-4 through 4-6 present the production reports for the fall preassessment 
contact for original sample and transfer schools. 
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Table 4-4. Fall preassessment contact productivity report, by week: School year 2001-02 



Week 


Week 

ending 


Total schools completed 


Original 


Transfer 


1 


10/10/01 


228 


158 


70 


2 


10/17/01 


581 


375 


206 


3 


10/24/01 


964 


541 


423 


4 


10/31/01 


1,317 


691 


626 


5 


11/7/01 


1,532 


760 


772 


6 


11/14/01 


1,759 


804 


955 


7 


11/21/01 


1,963 


849 


1,114 


8 


11/28/01 


2,160 


878 


1,282 


9 


12/6/01 


2,216 


885 


1,331 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade 
data collection, school year 2001-02. 



Table 4-5. Fall preassessment contact production report — original schools: School year 2001-02 



Week 

Week# ending 


Response 

rate 

(percent) 


Total 

number 


Out of 
scope 


In scope 


No 

action Pending Complete 


Refusal 


Not 
contacted 
Final within field 
refusal period 


1 


10/3/01 


7.3 


954 


7 


947 


730 


148 


69 


0 




2 


10/10/01 


20.1 


954 


24 


930 


434 


307 


187 


2 




3 


10/15/01 


32.3 


954 


32 


922 


264 


358 


298 


2 




4 


10/17/01 


41.6 


954 


38 


916 


132 


400 


381 


3 




5 


11/1/01 


76.8 


954 


54 


900 


3 


200 


691 


6 




6 


11/15/01 


89.4 


956 


57 


899 


5 


82 


804 


8 




7 


11/28/01 


96.4 


956 


55 


901 


0 


21 


869 


10 


1 


8 


11/29/01 


97.3 


956 


54 


902 


0 


13 


878 


9 


2 


9 


12/6/01 


98.1 


956 


54 


902 


0 


0 


885 


6 


5 6 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 



Table 4-6. Fall preassessment contact production report — transfer schools, by week: School year 2001-02 



Week 

Week # ending 


Response 

rate 

(percent) 


Total 

number 


Out of 
scope 


In scope 


No 

action 


Pending 


Complete 


Refusal 


Final 

refusal 


Not 
contacted 
within field 
period 


1 


10/3/01 


0.5 


1,410 


7 


1,403 


1,332 


64 


7 


0 






2 


10/10/01 


5.2 


1,410 


30 


1,380 


1,195 


113 


72 


0 






3 


10/15/01 


9.0 


1,410 


72 


1,338 


1,028 


189 


121 


0 






4 


10/17/01 


16.0 


1,410 


124 


1,286 


810 


269 


206 


1 






5 


11/1/01 


50.0 


1,585 


334 


1,251 


338 


277 


625 


11 






6 


11/15/01 


58.1 


2,093 


452 


1,641 


453 


219 


954 


15 






7 


11/28/01 


69.1 


2,259 


496 


1,763 


187 


150 


1,219 


19 




188 


8 


11/29/01 


72.9 


2,262 


503 


1,759 


96 


71 


1,282 


23 




287 


9 


12/6/01 


75.5 


2,268 


506 


1,762 


0 


0 


1,331 


10 


12 


409 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 



4 . 4.4 



Tracing Activities Between Fall- and Spring-Third Grade Data Collection 



In order to ensure that as many of the sampled children as possible were contacted in the 
spring, locating efforts continued in January and February 2002. Staff in Westat’s Telephone Research 
Center (TRC) traced the 781 children who could not be located during the preassessment school contact 
phase. TRC staff also used the Internet, telephone directories, and other means to locate these children 
and their households. When children and/or households were found, the new school and contacting 
information was entered into the tracing system database for fielding in the spring. Table 4-7 presents the 
results of this effort. See section 4.5.4 for more details about children who changed schools between first 
and third grade. 



Table 4-7. Results of the Telephone Research Center’s locating efforts for cases with a status of 
unlocatable at the end of the fall preassessment contact period: School year 2001-20 



Result 


Number 


Percent 


Total cases worked 


781 


100.0 


Located and entered into database 


307 


39.3 


Unlocatable 


426 


54.5 


Ineligible 


27 


3.5 


Final refusal 


16 


2.0 


Partially located 


3 


0.4 


Unable to locate due to language barrier 


2 


0.3 



NOTE: “Unlocatable” means that the children and their households could not be found using the available tracing and locating strategies; 
Children were “ineligible” for this round of data collection because they were deceased, had moved out of the data collection area, or were not in 
the sample of movers to be followed; “final refusal” means that the child’s family indicated that they did not want to participate; “partially 
located” means that the tracing and locating effort yielded some information about the child, but not enough to definitively locate the child; 
“unable to locate due to language barriers” means that the household language was not English and no staff were available who were bilingual in 
that language to communicate with the householder. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



A mailing to the post office requesting change of address information for sampled 
households was also conducted in November 2001. Westat mailed the requests for 17,472 eligible 
households. Table 4-8 presents the results of that mailing. 



4-31 







Table 4-8. Numbers and percents of the Post Office mailing results: November 2000 



Result 


Total 


Percent 


Mail is delivered to address given 


14,778 


84.6 


New/Boxholders street address 


1,510 


8.6 


Not known at address given 


633 


3.6 


Moved, left no forwarding address 


242 


1.4 


No such address 


93 


0.6 


Other' 


216 


1.2 



* These were cases where the post office could not provide a status on an address, but did not provide an explanation for why they could not. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 kindergarten, first, and third grade data collections, school years 1998-99, 1999-2000, and 2001-02. 



4.5 Spring-Third Grade Data Collection 

All children who were assessed during the base year or for whom a parent interview was 
completed in the base year were eligible to be assessed in the spring-third grade data collection. 
Eligibility for the study was not dependent on the child’s current grade; that is, children were eligible 
whether they were promoted to third grade, retained in an earlier grade, or promoted to a higher grade 
(e.g., fourth grade). 

As in previous rounds of data collection, the field staff were organized into work areas, each 
with a data collection team consisting of one field supervisor and three or more assessors/interviewers. 
The data collection teams were responsible for all data collection activities in their work areas; they 
conducted the direct child assessments and the parent interviews, distributed and collected all school and 
teacher questionnaires, and completed school facilities checklists. The majority of field staff members in 
third grade were continuing from previous rounds of data collection; a few new staff were hired in areas 
where no experienced ECLS-K staff lived. 



4.5.1 Spring Preassessment Activities 

Based on the information collected in the fall of 200 1 , packets of hard-copy teacher and 
school administrator questionnaires were assembled and mailed to schools in February 2002, along with 
letters confirming the scheduled visits to the school. Teachers and school administrators were asked to 
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either complete the questionnaires for pickup on assessment day, or to return the questionnaires in a 
Federal Express mailer that was provided in the packet. Letters and descriptive project summary sheets, in 
both English and Spanish, were also mailed to parents at this time reminding them of the spring-third 
grade data collection activities. (See appendix El: Letters and Descriptive Summary Sheets of the Third 
Grade Data collection [English and Spanish].) 

Most preassessment contact with schools was by telephone starting in March 2002. Field 
supervisors contacted the school to confirm the assessment date and the receipt of the hard-copy 
questionnaires and to arrange for space to conduct the assessments. This follow-up call to the schools was 
essentially to confirm the logistics for the assessments. 



4.5.2 Conducting the Direct Child Assessments 

The direct child assessments were conducted from late March through June 2002, the same 
time of year as in prior spring data collections. The school coordinator set the assessement date with a 
supervisor based on the school’s schedule. Approximately 91 percent of the assessments were completed 
in April and May and 9 percent were completed in June. There was no evidence that certain types of 
children or schools were assessed early or late in the field period. Table 4-9 presents the weekly 
completion rates for the child assessments. In year-round schools, assessment teams made multiple visits 
to the school, visiting when each track was in session to assess the sampled children. There were 617 (2.6 
percent) sampled children in year-round schools. 

The direct child assessments were usually conducted in a school classroom or library. Before 
conducting the assessments, field supervisors and assessors set up the room for the assessments. They 
followed procedures for meeting children that were agreed upon during the preassessment contact with 
the school. Each child was signed out of his or her classroom prior to the assessment and signed back into 
the classroom upon the conclusion of the assessment. When scheduling schools in the fall, an attempt was 
made to conduct the direct child assessments at about the same point in time from the beginning of school 
year and the end of the year to increase the chances that exposure to instruction was about the same for all 
children. The third grade direct child assessments averaged 94 minutes in length. 
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Table 4-9. Weekly completion of the child assessments: School year 2001-02 



Week # 


Week 

starting 


Completed 

assessments* 


Cumulative 

completed 

interviews 


Cumulative 
Percent 
of total 


1 


4/3/02 


1,165 


1,165 


8.0 


2 


4/10/02 


1,339 


2,504 


17.3 


3 


4/17/02 


1,113 


3,617 


24.9 


4 


4/29/02 


3,213 


6,830 


47.1 


5 


5/1/02 


421 


7,251 


50.0 


6 


5/9/02 


2,067 


9,318 


64.2 


7 


5/17/02 


1,776 


11,094 


76.4 


8 


5/24/02 


1,676 


12,770 


88.0 


9 


5/31/02 


484 


13,254 


91.3 


10 


6/7/02 


634 


13,888 


95.7 


11 


6/17/02 


286 


14,174 


97.7 


12 


6/24/02 


159 


14,333 


98.8 


13 


6/28/02 


106 


14,439 


99.5 


14 


7/3/02 


63 


14,502 


100.0 



' Based on field production reports. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Table 4-10 displays the number of completed child assessments for each round of data 
collection, including spring-third grade. All of the assessments in spring-third grade were completed in 
English. The majority of the assessments (74.6 percent) were completed in original schools, although the 
number of assessments in transfer schools has grown at each data collection point. About one quarter of 
the sample was assessed in transfer schools in spring-third grade. Assessments of homeschooled children, 
or children in nonparticipating transfer schools, were held at the parent’s home or another location, such 
as a library, of the parent’s choosing. 
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Table 4-10. Completed child assessments by round of data collection: School years 1998-99, 1999-2000, and 2001-02 





F all-kindergarten 


Spring-kindergarten 


Fall-first grade 


Spring-first grade 


Spring-third grade 


Category 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Child assessments completed 


19,147 


100.0 


19,987 


100.0 


5,297 


100.0 


16,622 


100.0 


14,502 


100.0 


in English 


17,019 


88.9 


18,342 


91.8 


4,848 


91.5 


15,460 


93.0 


13,565 


93.5 


In Spanish 


1,008 


4.3 


724 


3.6 


176 


3.3 


286 


1.7 


N/A 


N/A 


In other language 


410 


2.1 


229 


1.1 


33 


0.6 


37 


0.2 


N/A 


N/A 


With accommodation^ 


515 


2.7 


579 


2.9 


195 


3.7 


761 


4.6 


814 


4.6 


Excluded 


88 


0.5 


70 


0.4 


28 


0.5 


47 


0.3 


74 


0.5 


Partial complete 


107 


0.6 


43 


0.2 


17 


0.3 


31 


0.2 


49 


0.3 


Child assessments completed 


19,147 


100.0 


19,987 


100.0 


5,297 


100.0 


16,622 


100.0 


14,502 


100.0 


Original sampled school 


19,147 


100.0 


19,463 


97.4 


4,867 


91.9 


14,830 


89.2 


10,820 


74.6 


Transfer school 


0 


0.0 


524 


2.6 


430 


8.1 


1,792 


10.8 


3,682 


24.4 



* The term accommodation in this table is the field operational definition of accommodation, which includes the wearing of glasses and hearing aids. These type of aids were systematically tracked to 
ensure that every child had the same chance at a successful assessment. With this information, assessors could prompt a child, for example, to get her glasses before being assessed. 



NOTE: This table reflects final production numbers prior to statistical adjustment. This table does not include children who were subsampled out in fall- and spring-first grade and spring-third grade (see 
section 4.5.4.) These numbers should not be used to estimate student mobility. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) kindergarten, first, and third grade data 
collection, school years 1998-99, 1999-2000, and 2001-02. 





Approximately 1 percent of participating children in third grade required accommodations or 
were excluded from the direct child assessments (see section 4.4.2. 3). Children were excluded from the 
direct assessments because of a disability (e.g., blindness or deafness) that was not accommodated by the 
ECLS-K direct assessments or because their lEP prevented their participation in assessments or required 
an accommodation not offered in the assessments. Accommodations offered in the assessments were as 
follows: alternative setting, scheduling, or timing; health care aide present; or the use of an assistive 
device. Table 4-11 presents the number of children excluded from or requiring an accommodation to the 
direct child assessment procedures in the spring of third grade. 



Table 4-11. Number of children excluded from or accommodated in the spring-third grade direct child 
assessments: School year 2001-02 



Category 


Number of children 


Exclusions 


Excluded for disability 


74 


Accommodations ' 


Alternative setting accommodation 


33 


Scheduling/timing accommodation 


65 


Health care aide present 


6 


Assistive device 


4 



' The term accommodation in this table includes only those accommodations offered during the assessment such as an alternative setting. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school years 2001-02. 



4.5.3 Conducting the Parent Interview 

Parent interview procedures mirrored those of the base year and first grade. The parent 
interview was administered primarily by telephone interview using CAI from March to July 2002. (See 
appendixes E2 and E3 for the English and Spanish versions of the interview.) Sixteen percent of the 
parent interviews were completed in March, 54 percent were completed in April and May, and 30 percent 
were completed in June or later. Table 4-12 presents the weekly completion of parent interviews. The 
parent interview averaged 62 minutes. 
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Table 4-12, Weekly completion of parent interviews, spring-third grade: School year 2001-02 



Week 


Week 

starting 


Completed 

Interviews’ 


Cumulative 

completed 

interviews 


Cumulative 
Percent 
of total 


1-4 


3/4-31/02 


2,165 


2,165 


16.3 


5 


4/3/02 


2,518 


2,518 


19.0 


6 


4/10/02 


1,084 


3,602 


27.2 


7 


4/17/02 


763 


4,365 


32.9 


8 


4/29/02 


1,612 


5,977 


45.1 


9 


5/1/02 


175 


6,152 


46.4 


10 


5/9/02 


941 


7,093 


53.5 


11 


5/17/02 


708 


7,801 


58.9 


12 


5/24/02 


1,055 


8,856 


66.8 


13 


5/31/02 


485 


9341 


70.5 


14 


6/7/02 


822 


10,163 


76.7 


15 


6/17/02 


654 


10,817 


81.6 


16 


6/24/02 


555 


11,372 


85.8 


17 


6/28/02 


578 


11,950 


90.2 


18 


7/3/02 


293 


12,243 


92.4 


19 


7/12/02 


172 


12,415 


93.7 


20 


7/19/02 


331 


12,746 


96.2 


21 


7/29/02 


289 


13,035 


98.4 


22 


8/2/02 


214 


13,249 


100.0 



' Based on field production reports from the parent interview. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 third grade data collection, school year 2001-02. 



As in previous rounds of data collection, the parent interview was conducted in person if the 
respondent did not have a telephone and in other languages as needed. Table 4-13 contains the number of 
parent interviews per round, including spring-third grade by mode of administration and language used. In 
third grade, only 2.4 percent of all completed parent interviews were conducted in person; 7.2 percent of 
all completed parent interviews were conducted in a language other than English with 94.8 percent of 
completed non-English interviews conducted in Spanish. 
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Table 4-13. Number and pereent of completed parent interviews by data collection mode, language, and wave of data collection: School years 
1998-99, 1999-2000, and 2001-02 





F all-kindergarten 


Spring-kindergarten 


Fall-first grade 


Spring-first grade 


Spring-third grade 


Category 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Total parent interviews 
completed 


17,997 


100.0 


18,907 


100.0 


5,073 


100.0 


15,576 


100.0 


13,504 


100.0 


Mode 






















In person 


618 


3.4 


619 


3.3 


211 


4.2 


456 


2.9 


319 


2.4 


By phone 


17,379 


96.6 


18,288 


96.7 


4,862 


94.8 


15,120 


97.1 


13,185 


97.6 


Language 






















In English 


17,379 


96.6 


17,482 


92.5 


4,717 


93.0 


14,319 


91.9 


12,416 


91.9 


In language other 
than English 






















In Spanish 


618 


3.4 


1,321 


7.0 


351 


6.9 


1,071 


6.9 


932 


6.9 


In other language 


0 


0 


81 


0.4 


0 


0 


75 


0.5 


41 


0.3 


Partial complete' 


0 


0 


23 


0.1 


5 


0.1 


111 


0.7 


115 


0.9 



’ A partial complete is parent interview in which some, but not all sections have been completed. To be considered a partial complete, at least the first three sections (Introduction, Parent Involvement, 
and Family Structure) had to have been completed. 

NOTE: This table reflects final production numbers. Detail may not sum to totals due to rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) kindergarten, first, and third grade 
data collections, school years 1998-99, 1999-2000, and 2001-02. 



In spring third grade data, 21 percent of the parent cases were classified as final 
nonresponse. As in most field studies, the primary reasons for final nonresponse were parents who could 
not be located and parents who refused to complete the interview; 31.4 percent of the parent cases were 
unable to be located, 54.1 percent were refusals and 14.5 percent were other nonresponse (e.g. language 
barrier). Because the ECLS-K is a longitudinal study, nonresponse builds over time. Spring third grade 
data collection began with 6.4 percent of the parent cases classified as either could not be located (66 
percent of initial nonresponse cases) or refusal cases (34 percent of initial nonresponse cases). During 
data collection additional cases could not be located (6 percent of parent cases) and refused (3 percent of 
parent cases). Field staff were somewhat successful with completing the initial nonresponse cases; 21.5 
percent of initial could not be located were found and interviewed and 30 percent of initial refusal cases 
were convinced to participate. 

A special effort to build parent interview response rates was conducted from July 5 to 31, 
2002, and yielded an additional 7.3 percentage points to the response rate. Almost 8 percent (7.7 percent) 
of the parent interviews were not completed because of locating problems. Almost 7 percent (6.6 percent) 
of the parent interviews were not completed because of refusal, either regular (2.9 percent) or hard (3.7 
percent). Hard refusals from previous rounds were not fielded for data collection during third grade. 



4.5.4 Conducting Data Collection on Children Who Withdrew From Their Previous Round 

School 

While contacting schools, field supervisors asked school coordinators to identify children 
who had withdrawn from the school since the spring of first grade. School staff were asked whether they 
knew the name and address of the school to which the child transferred, as well as any new information 
about the child’s household address. For the children who had moved from their spring-first grade school 
and were not part of the sample to be followed, information was collected only from the school personnel 
and not parents. For children who had withdrawn from their spring-first grade school and were identified 
to be followed (i.e., were part of the sample of movers), supervisors also consulted parents and other 
contacts for information on the children’s new school. This information was entered into the FMS and 
processed at Westat for data collection. 

As previously mentioned, 4,679 movers were identified during the fall preassessment 
contact. During spring-third grade, an additional 1,628 movers were identified (3,582 movers were 
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identified since kindergarten and fiagged to be followed for data collection). The identification of 75 
percent of the third grade movers during the fall contact accomplished two important goals: (1) allowed 
additional time to trace movers; and (2) reduced the burden of tracing a large number of movers during 
the spring-third grade data collection. 



Table 4-14 presents the final status of the children who were identified as movers in third 
grade; a total of 9,889 children were identified as having transferred from the school in which they were 
enrolled when they were sampled in kindergarten. Of the 9,889 children identified as movers in spring- 
third grade, 5,668 children were selected to be followed and were followed (57.3 percent of total movers). 
The remaining 4,22 1 mover children were ineligible for this round of data collection because they moved 
out of the country, were deceased, or were subsampled out; no child assessments or parent interviews 
were conducted for these children. 

Table 4-14. Number of children who moved by completion category: School years 1998-99, 

1999-2000, and 2001-02 





Spring-kindergarten 


Spring- first grade 


Spring-third grade 


Category 


Number 

of 

children 


Percent 


Number 

of 

children 


Percent 


Number 

of 

children 


Percent 


Total movers 


1,568 


100 


5,759 


100 


9,889 


100 


Ineligible* 


16 


1 


2,893 


50.2 


4,221 


42.7 


Did not follow^ 


0 


0 


2,848 


98.4 


4,102 


97.2 


Moved to outside of U.S.^ 


16 


1 


44 


1.5 


117 


2.7 


Deceased^ 


0 


0 


1 


<1 


2 


<1 


Eligible and followed* 


1,552 


98.9 


2,866 


49.8 


5,668 


57.3 


Completed assessment^ 


510 


32.8 


1,792 


62.5 


3,682 


65.0 


Unlocatable^ 


501 


32.3 


271 


9.5 


607 


10.7 


Nonsampled primary 
sample unit^ 


282 


18.2 


454 


15.8 


871 


15.4 


Nonresponse 


259 


16.7 


349 


12.2 


508 


8.9 



’ Percent based on total movers. 

^ Percent based on ineligible children. 

^ Percent based on eligible children. 

SOURCE; U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 (ECLS-K) kindergarten, first, and third grade data collections, school years 1998-99, 1999-2000, and 2001-02. 
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Different data collection strategies were followed for children who moved, depending on 
where they moved and the status of their new school. Data collection was attempted for children who 
moved and were flagged as “follow” in spring-third grade in the following ways: 



■ Children moving into cooperating schools were treated as nonmovers. Data collected 
for children moving into cooperating base year sampled schools included the child 
assessments in the school, school administrator questionnaire, school fact sheet, 
regular and/or special education teacher questionnaires, facilities checklist, and 
student records abstract forms, and parent interview. 

■ Data collected for children moving into nonsampled schools in base year cooperating 
districts included the child assessments in the school, school administrator 
questionnaires, school fact sheet, regular and/or special education teacher 
questionnaires, and student records abstract forms, if the school agreed to participate. 
If school permission was not obtained, the assessments were conducted in the home 
and no school or teacher data were collected. Parent interviews were attempted for all 
children. 

■ For children moving into transfer schools that refused, schools in sampled districts 
that refused, or originally sampled schools that were ineligible when sampled because 
they did not have kindergarten classes, the direct child assessments were conducted in 
the home. No school or teacher data were collected. Parent interviews were attempted 
for all children. 

■ For children moving into schools in nonsampled districts or dioceses: 

If the school was within the primary sampling unit (PSU), data collected 
included the child assessments in the school, school administrator 
questionnaire, school fact sheet, regular and/or special education teacher 
questionnaires, facilities checklist, and student records abstract forms, if school 
permission was obtained. If school permission was not obtained, the 
assessments were conducted in the home and no school or teacher data were 
collected. Parent interviews were attempted for all children. 

If the school was outside the PSU, no child, school, or teacher data were 
collected. The parent interview was still attempted. 

■ For children who were not enrolled in school in the spring (including children who 
were home schooled), data collected included the child assessments in the home if the 
child was in the sampled PSU. If the child was outside the sampled PSU, no child 
assessment or school or teacher data were collected. Parent interviews were attempted 
for all children. 

Of the children who moved in third grade and were selected to be followed, 15.4 percent 
moved into a school outside the PSU and 10.7 percent of the movers could not be located. Assessments 
were completed for 65 percent of the movers who were followed in the spring-third grade data collection 
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and parent interviews were completed for 68 percent of these children. Table 4-15 presents the third grade 
movers by school and district status. 

Table 4-15. Children moving in third grade: School year 2001-02 



Category 


Number 


Percent 


Total movers eligible and followed 


5,668 


100 


Moving into cooperating schools 


93 


17.5 


Moving into nonsampled schools 


2,735 


48.3 


Moving into schools/districts that refused 


372 


6.6 


Homeschooled 


90 


1.5 


Moving outside the primary sampling unit 


871 


15.4 


Moving — could not be located 


607 


10.7 


Moving — final nonresponse 


900 


15.9 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



4.5.5 Teacher and School Data Collection 

Data were collected from school administrators, regular classroom teachers, and special 
education teachers from March through June 2002. 

The school and teacher questionnaires were mailed to the school coordinators in February 
2002. This schedule allowed 2 months of additional time for these respondents to complete and return the 
instruments to Westat. Using the child-teacher linkage information collected in the fall, a packet of 
questionnaires was assembled for each regular and special education teacher. The regular teacher packet 
included a cover letter, a sheet explaining the study and its goals, teacher questionnaire A, teacher 
questionnaire B, and teacher questionnaire C for each student who had been linked to the teacher in the 
fall. The special education teacher packet contained a cover letter and summary sheet, special education 
teacher questionnaire A, and special education teacher questionnaire B for each sampled student linked to 
the teacher. Packets were bundled together by school and mailed to the school coordinator for 
distribution. If the school and/or teacher and school administrator were not identified in the fall advance 
contact, then the supervisor gathered the relevant information during the preassessment call in the spring 
and mailed the packets at that time. During their visits to the schools, field supervisors also completed a 
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facilities checklist for each sampled school. (See appendix E4: Cover Letters to School Coordinators and 
appendix E5: Hard-Copy School and Teacher Questionnaires.) 

Field supervisors began prompting for the return of questionnaires when they contacted 
schools to confirm the assessment schedule. During the field period, field supervisors followed up with 
school administrators and teachers by telephone and visits to the schools to prompt for the return of the 
questionnaires and collected completed questionnaires to return to Westat. 

In May 2002, the field supervisors were instructed to conduct intensive followup for missing 
school administrator questionnaires for schools with high minority student populations. There were 99 
such schools identified. As a result of the followup efforts by the field supervisors, 75 of the 99 schools 
(76 percent) completed and returned the school administrator questionnaires. 



4.5.5.1 Hard-Copy Data Retrieval 

Data retrieval involved collecting missing items for questionnaires that were otherwise 
complete. A list of critical items by questionnaire was finalized in April 2002. Exhibit 4-13 presents the 
critical items by questionnaire. 



Exhibit 4-13. Critical items by questionnaire type: School year 2001-02 



Questionnaire type 


Critical items 


School administrator questionnaire 


Items 1, 3, 13, 14, 44, 33, 34a, 34b, 50 


Teacher questionnaire A 


Items 1,2,4, 5, 17,38, 40 


Teacher questionnaire B 


(If item 17 or 22 is missing): Items 10, 11, 12, 13, 14, 15 


Teacher questionnaire C 


Items 1, 2, 13, 14 


Special education teacher questionnaire A 


Items 7, 10, 12 


Special education teacher questionnaire B 


Items 3, 5, 1 1 


Student records abstract 


Items 6, 7, 8, 10 


School fact sheet 


Items 1, 2, 3, 4, 6 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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As they collected complete questionnaires, field supervisors reviewed the critical items for 
each questionnaire to ensure that such items were completed before sending the questionnaires to Westat. 
Field supervisors attempted two types of data retrieval: 



■ Scanning for missing critical items in questionnaires that they collected directly from 
the school and teachers, and attempting to retrieve the missing items; 

■ Completing Data Retrieval Forms for questionnaires the schools and teachers returned 
directly to Westat, and attempting to retrieve missing items. 



4.5.S.2 Scan for Missing Critical Items 



Field supervisors were asked to scan questionnaires for missing critical items, that is, items 
identified as especially important to analysts. The critical items were packaged for the field supervisors in 
a comprehensive list (see appendix E6: Third Grade Questionnaires — Critical Item List). In the Third 
Grade Questionnaire — Critical Item List the procedures were listed as follows: 



■ For each questionnaire, the critical items were listed along with the rule for attempting 
data retrieval for each item. The color of the questionnaire cover was included with 
the name of the questionnaire. 

■ As field supervisors collected completed questionnaires, they scanned the 
questionnaires for missing critical items using the Critical Item List. They only 
scanned questionnaires for the critical items listed in the Critical Item List. They were 
instructed not to attempt to edit the questionnaire or change responses. 

■ As field supervisors identified missing critical items, they affixed a tape flag to the 
questionnaire page so they could quickly flip to the item when they reviewed the 
questions with the respondent. 

■ The student records abstract (SRA) was to be completed after the school year ended. 
If the field supervisor collected the completed SRAs, they were instructed to scan 
them for completeness and attempt to retrieve any missing data. 



4.5.S.3 Retrieving Missing Critical Items from Questionnaires Received in the Home Office 



As described earlier, questionnaires were mailed to schools and teachers in February 2002. 
School staff were given the option of either returning the questionnaires via an enclosed Federal Express 
mailer or holding the questionnaires until the field supervisors visited the schools. By the time the critical 
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item list was finalized 6 weeks into the field period, Westat had received more than 1 1,000 questionnaires 
directly from the school respondents. About 1 1 percent of the receipted questionnaires required retrieval 
of missing critical items, with the majority of the data retrieval required for the school administrator 
questionnaire and teacher questionnaire A. 

Receipt control staff scanned the questionnaires for critical items upon receipt (see chapter 5 
for a more detailed discussion of this process). When a critical item was identified as missing a response, 
a Data Retrieval Form was completed. There was a Data Retrieval Form for each type of hard-copy 
questionnaire (see appendix E7: Data Retrieval Questionnaires). The Data Retrieval Form for each 
questionnaire contained only the critical items and the receipt staff highlighted missing critical item(s) on 
the Data Retrieval Form. 

The Data Retrieval Forms were mailed to field supervisors to attempt to retrieve missing 
critical items. Each form had a label on the cover that identified: 

■ The questionnaire type; 

■ The supervisor name, region, and work area; 

■ The school name, ID, address, and phone number; 

■ The school coordinator name (only on the school administrator questionnaire and the 
school fact sheet); 

■ The teacher name and ID number (only on teacher questionnaires A, B, and C and 
special education teacher questionnaires A and B); 

■ The child name and ID number (only on teacher questionnaire C, special education 
teacher questionnaire B and student records abstracts); and 

■ The respondent name and ID number (only on student records abstracts). 

Field supervisors were instructed to review each Data Retrieval Form to determine which 
teachers and/or school staff they should contact, and for which critical items they were to attempt data 
retrieval. 
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4.5.S.4 



In-Field Data Retrieval Attempts 



Field supervisors attempted to retrieve missing critical items and missing questionnaires in 
the schools in their assignments. They scheduled their retrieval efforts for the day the assessments were 
scheduled and attempted to find the respondents in person. Otherwise, they attempted these retrieval 
attempts by telephone. Field supervisors recorded any changes to missing critical items in blue pencil in 
the questionnaire. If the respondent did not know the answer, they recorded “DK” by the item; if the 
respondent refused to answer, they recorded “RF” by the item. Field supervisors recorded the status of the 
questionnaire as one of the following: 

■ Questionnaire complete with no missing critical items: no data retrieval required. 

■ Questionnaire is missing one or more critical items: data retrieval required; one or 
more critical items collected. 

■ Questionnaire is missing one or more critical items: data retrieval required; no critical 
items were collected. 

■ Questionnaire refused: unit nonresponse. 

Table 4-16 presents the final results of the critical item retrieval. 

4.5.S.5 School Fact Sheet and Student Records Abstract Followup 

To improve response rates on the school fact sheet and student records abstract, a telephone 
followup effort was conducted in fall 2002. The school fact sheet and student records abstracts included 
items that were not time-sensitive that school staff could abstract from records. A package was mailed in 
early September 2002 to all nonrefusing schools with outstanding school fact sheets or student records 
abstracts with a request to complete and return these questionnaires. The package included a customized 
letter and questionnaires labeled with the student name and ID number (see appendix F: Followup Letters 
and Questionnaires [School Fact Sheets and Student Records Abstracts]). 

The experience in prior rounds of data collection was that prompting schools to return the 
questionnaires in the summer was ineffective. The students records abstracts were particularly 
problematic with respect to retrieval because the school staff could not complete the form until the school 
year ends. In the first grade, mail prompting for unit nonresponse began in early June. Rarely were school 
staff physically in the school at that time. Sometimes the principal was present, but there were typically 
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Table 4-16. Total number and percent of the final results of critical item retrieval: School year 2001-02 





Cases needing no data retrieval 


Cases needing data 
retrieval 


Data retrieval completed 


Refused 




Questionnaire type 


Total 

expected' 


Number 


Percent^ 


Number Percent^ 


Number 


Percent 


Number Percent^ 


School administrator 


2,852 


736 


25.8 


1,141 


40.0 


728 


63.8 


975 


34.2 


School fact sheet 


2,852 


1,785 


62.6 


103 


4.0 


78 


75..7 


964 


33.8 


Teacher questionnaire A 


6,189 


3,652 


59.0 


750 


1.2 


565 


75.3 


1,787 


28.9 


Teacher questionnaire B 


6,189 


4,269 


68.9 


122 


2.0 


77 


63.1 


1,798 


29.1 


Teacher questionnaire C 


14,548 


11,428 


78.6 


387 


3.0 


298 


77.0 


2,733 


18.9 


Special education 
teacher questionnaire A 


891 


640 


71.8 


10 


1.0 


6 


60.0 


241 


27.0 


Special education 
teacher questionnaire B 


1,177 


855 


72.6 


32 


3.0 


22 


68.8 


290 


24.6 


Student records abstract 


14,528 


9,133 


62.9 


1,136 


<1 


218 


19.1 


4,259 


29.3 



This column represents the number of questionnaires expected from school and teacher respondents based on schools and teachers linked to sampled children. 

^ Percent based on number expected. 

^ Percent based on number complete, needing data retrieval. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, school year 
2001 - 02 . 



no teachers or secretaries. In first grade, telephone prompting for unit nonresponse began in August 2000, 
but school staff often reported they were too busy to respond until mid- September. Based on the 
kindergarten and first grade experience, for third grade no attempt to retrieve school fact sheets or student 
records abstracts for unit nonresponse was made in summer 2002. A telephone prompting effort was 
mounted through Westat’s TRC, starting in mid-September, after the 2002-03 school year started. School 
staff were prompted by telephone through October 2002 to return the questionnaires and abstracts. The 
hard-copy followup increased child-level response rates for the school fact sheet by 10 percent and the 
student records abstract by 12 percent. 



4.5.6 Incentives in the ECLS-K 



In order to gain respondent cooperation and ensure participation throughout the various data 
collection phases of the ECLS-K study, various incentives were offered. The type of incentive, monetary 
or nonmonetary, depended on whether the respondent was a sampled child, parent, teacher, or school. 
Exhibit 4-14 delineates the types of incentives used in the ECLS-K. 



Exhibit 4-14. Types of incentives used in the ECLS-K: School year 2001-02 



Respondent 


Incentive 


Child 


■ 


ECLS-K Pen 




■ 


Birthday card 


Parent 


■ 


Newsletters 




■ 


ECLS-K Post-It Notepad 




■ 


Trade books 


Teacher — original schools 


■ 


$7 for each completed child- level questionnaire 


Teacher — transfer schools 


■ 


$20 for up to 3 completed child- level questionnaires 




■ 


$7 for each additional child-level questionnaire 


School — original 


■ 


$200 for cooperating schools 




■ 


$7 for each completed student records abstract 


School — transfer 


■ 


$7 for each completed student records abstract 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Child Incentives. Children were given a small token at the end of the assessment to thank 
them for their cooperation in completing the assessment. In the spring-third grade, they were given a pen 
with the phrase “I love ECLS-K” and the toll-free number for the respondent to call. The pen was blue 
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and red with white lettering. In addition, each month Westat mailed birthday cards to children whose 
birthdays fell within that month. Children were sent birthday cards throughout the calendar year, not just 
during the school year. By mailing these cards, children were not only thanked again for their 
participation, but parents are also reminded about the study. These periodic reminders are important in a 
longitudinal study, in which respondents may become apathetic toward the study during later rounds. Not 
only do these reminders encourage respondent participation, but they help the home office update 
addresses of families that have moved. 

Parent Incentives. In the spring of 2000, shortly before the beginning of first grade data 
collection, a newsletter about the study was published and mailed to parents. The newsletter served to 
update respondents on the initial findings from the fall of kindergarten year and inform them about the 
future rounds of data collection. Reading and mathematics skills, social skills, and child care were a few 
of the topics discussed. Not only did the newsletter update parents on the findings of the study and 
highlight its importance, but it also was an incentive for future rounds of participation. Respondents were 
able to see the results of their participation in the study. 

Parents received an incentive for participating in the parent interview. At the end of the field 
period, thank-you letters were generated for every respondent who completed a parent interview. The 
letters, along with an ECLS-K Post-It notepad, were mailed to the respondent. Letters translated into 
Spanish were mailed to those respondents who completed the interview in Spanish. (See appendix Gl: 
Parent Thank Y ou Letters [English and Spanish]) 

Teacher Incentives. Teachers in original ECLS-K schools were asked to complete 
individual ratings for the sampled children in their classrooms, and they were reimbursed $7 for each 
child rating (teacher questionnaire C and special education teacher questionnaire B) they completed. 
Teachers in transfer schools were also asked to complete individual ratings for the sampled children in 
their classrooms, and they were reimbursed $20 for up to 3 child ratings and $7 for each additional 
completed child rating. 

School Incentives. Original schools were also paid a monetary incentive for participating in 
the ECLS-K. Because school staff are often very busy and may not be aware of the benefits of 
cooperating, the cooperating original sample schools were remunerated $200 for participating. School 
staff who completed the student records abstract received $7 for each questionnaire completed. 
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All checks to schools, teachers, and school staff completing the SRAs were mailed weekly 
during the field period and were sent with thank-you letters. The checks to teachers and to staff 
completing the SRA were made out the individual who completed the forms. (See appendix G2: Thank 
You Letters for the School, Teacher, and Student Records Abstract Respondents.) 



4.6 Data Collection Quality Control 

The ECLS-K data are used by researchers to study children’s school experience and its 
relation to student outcomes, and by educators and policymakers to inform policy decisions. It is 
important that the information used by these groups is based on sound research practice and that 
considerable attention be paid to identifying potential sources of error, quantifying this error, and 
designing techniques to either reduce the error or minimize its impact on survey estimates. The work 
carried out in support of the ECLS-K includes a variety of activities that are directed toward ensuring that 
the data are of high quality. 



4.6.1 Maintaining Reliability on the Child Assessment 

To ensure that assessors maintained the standard that they achieved at training, assessors 
were observed by their supervisor in the field at 2 different points in time. The first observation was to be 
conducted within the first 2 weeks of the field period and the second observation 2 to 3 weeks after the 
first. The supervisor completed the Assessment Observation Form (see appendix HI: Assessment 
Observation Form), which rated the assessor on key areas of the assessment protocol. In the Assessment 
Observation Form, the supervisor simultaneously coded with the assessor those open-ended assessment 
items that required judgment by the assessor to determine whether the child’s answer was correct. At the 
end of the assessment period, after the child was escorted from the room, the supervisor and the assessor 
reviewed the assessor’s overall performance. The two also compared the way that they each handled the 
open-ended questions. If there were large discrepancies, they reviewed the QxQs for these items 
carefully. 
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4.6.1.1 



Assessment Observation Form 



The Assessment Observation Form had the names of the assessor and the supervisor, the 
case ID observed, the observation number, as well as the date the observation was conducted. The form 
had two sections: section 1 (shown in exhibit 4-15) was use by supervisors to rate the assessor on key 
overall skill areas, such as building rapport, using neutral praise, responding to behaviors presented by the 
child, pacing appropriately, and coaching. In section 1 the supervisor checked “No” for each skill area 
that the assessor did not demonstrate appropriately. 



Exhibit 4-15. Section 1 of the Assessment Observation Form: School year 2001-02 



Evaluator: As the assessment is administered, record whether or not the assessor successfully performed 
the following behaviors. Check “No” if the assessor makes 3 or 4 errors and needs to make improvements. 


SECTION 1 : RaoDort bulldlnq and worklnq with the child 


Q: Did the assessor... 


No 


1 . Establish rapport with the child? 




2. Use an appropriate pace with the child? 




3. Use neutral praise? 




4. Respond to behaviors presented by child? 




5. Avoid coaching the child? 





SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Section 2 listed specific questions from each routing and subdomain (e.g., mathematics) 
form. The instructions for completing section 2 are shown in exhibit 4-16. For each of the listed questions 
observed, the supervisor recorded both the child’s response and if the assessor did not demonstrate the 
specified required administration skills for that question. The required administration skills included 
reading questions verbatim, using appropriate probes, and using appropriate hand motions. For each 
question in which the supervisor observed that the assessor did not demonstrate the required 
administration skill(s), he or she checked a box, indicating which skill was not performed. 



4-51 














Exhibit 4-16. Instructions for Section 2 of the Assessment Observation Form: School year 2001-02 



SECTION 2: Specific Assessment Activities 

Supervisor/Evaluator: Code the items as the assessor administers the assessment. 

Code the chiid’s response as the item is administered. 

If the item requires probing, check the box if the assessor does not use the appropriate probe. 

Check the box in the “Verbatim” coiumn if the assessor does not read the item exactiy as worded on the 
screen. 

Check the box in the “Gesturing” column if the assessor does not use appropriate hand motions. 

For each validation code item, check the box in the “Validation” column if the response coded by the 
assessor is not what you have coded. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



As described earlier, during the observed assessment, the supervisor coded the child’s 
response to those open-ended assessment items that required judgment on the part of the assessor. These 
open-ended items were flagged in the CAI program for quality control review, and a screen showing how 
the assessor coded each of the questions was accessed at the end of the assessment for review with the 
supervisor. The supervisor and assessor compared their codes to these open-ended questions. For each 
item where the coding did not match, the supervisor recorded this discrepancy by checking the 
corresponding box for that question in the Assessment Observation Form. 

Exhibit 4-17 shows an example of a question from the Assessment Observation Form, which 
required the supervisor to evaluate the required assessment administration skills: reading verbatim, using 
appropriate gesturing, using appropriate probe, and accurately coding the child’s response to this open- 
ended question. The supervisor was instructed to check the boxes only if the assessor did not demonstrate 
the correct behavior. For all questions that are observed by the supervisor and listed in the Assessment 
Observation Form, the supervisor recorded if the assessor did not read verbatim or gesture appropriately. 
Supervisors only recorded if assessors did not use the appropriate probe for those questions that display 
probes (a box labeled “Assessor Used Inappropriate Probe?’’ was presented in the Assessment 
Observation Form). Also, the supervisors only recorded if their response coding did not match the 
assessors’ for those questions that are open-ended (a box labeled “Validation Code Did Not Match’’ was 
presented in the Assessment Observation Form). 
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Exhibit 4-17. Sample question from Assessment Observation Form: School year 2001-02 





DID NOT READ 
QUESTION 
VERBATIM 


GESTURED 

INAPPRO- 

PRIATELY 


VALIDATION 
CODE DID 
NOT MATCH 


M3. 300 Arnold made a star shape with these pattern blocks. (SWEEP 
PATTERN BLOCKS)** 

Here is a picture of the star shape that Arnold made. (POINT 
TO STAR) 

How many of each different shape of block did Arnold use to 
make his star shape? 

IF STUDENT RESPONDS WITH “5”, or “1 AND 4” PROMPT: 
“5/ 1 and 4 of which type/shape of block?” 

IF STUDENT RESPONDS ONLY WITH “SQUARE(S) AND 
TRIANGLE(S),” PROMPT: “How many squares and how many 
triangles?” 

CORRECT (1 SQUARE AND 4 TRIANGLES) 1 

INCORRECT 2 

REFUSED Ill 

DON’T KNOW 999 

ASSESSOR USED INAPPROPRIATE PROBE? □ 


□ 


□ 


□ 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



At the end of the child assessment, after the child was escorted from the room, the supervisor 
and the assessor reviewed the assessor’s overall performance. After discussing the ratings on section 1, 
the assessor accessed the quality control screen. The assessor and supervisor reviewed their codes for 
each open-ended question asked in section 2 of the Assessment Observation Form. 

The supervisor then scored the assessment observation using the scoring form shown in 
exhibit 4-18. In the first part of the form (Form A), the supervisor counted the number of check marks 
recorded in section 1 : Rapport Building and recorded that number in the appropriate row. The supervisor 
then recorded the number of check marks for each section of the assessment that was observed and 
entered those numbers in the appropriate box. Exhibit 4-19 shows an example of a completed Form A. In 
this example, the supervisor recorded one (1) check mark for section I, zero (0) check marks for Reading 
Routing, zero (0) check marks for Reading Blue, one (1) check mark for Math Routing, zero (0) check 
marks for Math Yellow, zero (0) check marks for Science Routing, and one (1) check mark for Science 
Red. The total check marks recorded was three (3). 
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Exhibit 4-18. Observation Scoring Form: School year 2001-02 



Observation Scoring 

Record the number of check marks from section 1: Rapport Building and section 2: Specific 
Assessment Activities in the appropriate boxes of Form A. 

Sum each row and record total in Row Totals column of Form A. 

Sum Row Totals column (including number of check marks from section 1) and record in Total 
Check Marks box of Form A. 



Form A: Checks Marks Recorded in Sections 1 and 2 



Assessment 


Routing Form 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 












Mathematics 












Science 












Number of check marks from section 1: Rapport Building: 




Total Check Marks: 





Step 4: Using Form B, circle the number in the form used for each assessment. For example, if the Red 

Form was used for reading, circle the number 20 in the Reading row under Red Form. Record 
each circled number in the Row Totals column of Form B. 

Step 5: Sum Row Totals column (including Possible points for Routing Forms and section T) and 

record in Total possible points box of Form B. 



Form B: Total Possible Points 



Assessment 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 


21 


24 


22 




Mathematics 


12 


16 


20 




Science 


9 


21 


20 




Possible Points for Routing Forms and section 1: Rapport Building: 


27 


Total Possible Points: 





Step 6: Use Proportion Correct Chart to determine the proportion correct and write that proportion in 

this box: I I 



Result: If Proportion correct is greater than or equal to .85, Assessor is Passed. 

If Proportion correct is between .70 and .85, Assessor requires Remedial Action (Complete 
Remedial Task Chart below) 

If Proportion correct is less than .70, Assessor Failed. 



Step 1: 

Step 2: 
Step 3: 



See note at end of exhibit. 



4-54 








Exhibit 4-18. Observation Scoring Form: School year 2001-02 — Continued 



Remedial Task Chart 

Circle the number of the task(s) needing improvement. Discuss remedial action with the assessor. 


TASK 


# 


REMEDIAL ACTION 


APPROPRIATE PACE 


1 


PRACTICE/REVIEW JOB AIDS 


AVOID COACHING 


2 


REVIEW THE JOB AIDS 


READ VERBATIM 


3 


PRACTICE READING FROM THE SCREEN 


GESTURING APPROPRIATEEY 


4 


PRACTICE GESTURING 


USE APPROPRIATE PROBES 


5 


REVIEW THE JOB AIDS/Q-BY-Qs 


VAEIDATION CODE DID NOT MATCH 


6 


REVIEW THE Q-BY-Qs 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Exhibit 4-19. Example of Form A: School year 2001-02 



Form A: Checks Marks Recorded in Sections 1 and 2 



Assessment 


Routing Form 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 


0 






0 


0 


Mathematies 


1 




0 




1 


Seienee 


0 


1 






1 


Number of eheek marks from section 1: Rapport Building: 


1 


Total Cheek Marks: 


3 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 third grade data collection, school years 2001-02. 



The supervisor then calculated the number of possible points for the sections of the 
assessment observed using the second section of the scoring form (Form B) (exhibit 4-20). Continuing 
with the example started earlier, the supervisor observed all of Reading Blue (22 possible points), Math 
Yellow (16 possible points), and Science Red (9 possible points). In addition, section I and all the 
Routing Forms contribute 27 possible points. Based on the sections of the assessment that the supervisor 
observed the total number of possible points for this assessor was 74. 



Exhibit 4-20. Example of Form B: School year 2001-02 



Form B: Total Possible Points 


Assessment 


Red Form 


Yellow Form 


Blue Form 


Row Totals 


Reading 


21 


24 


22 


22 


Mathematics 


12 


16 


20 


16 


Science 


9 


21 


20 


9 


Possible Points for Routing Forms and section 1: Rapport Building: 


27 


Total Possible Points: 


74 






SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 third grade data collection, school year 2001-02. 
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Finally, the supervisor determined the proportion correct by using the Proportion Correct 
Chart displayed in exhibit 4-21. Continuing the example, the supervisor found the column on the 
Proportion Correct Chart displaying the total possible points for the assessor based on the portions of the 
assessment observed (74) and the row on the Proportion Correct Chart displaying the total check marks 
the supervisor recorded for the trainee (3), and recorded the proportion from the corresponding box on the 
Proportion Correct Chart in the box under step 6 (exhibit 4-22) on the scoring form (.98). 

Once the supervisor completed the scoring, the assessor was rated as Passed, Remedial 
Action, or Failed. 

The field supervisors recorded their observations on the form and then reviewed the form 
with the assessor. The most frequent problems observed were not reading the items verbatim and 
inappropriate gesturing. Feedback was provided to the assessors on the strengths and weaknesses of their 
performance and, when necessary, remedial training was provided in areas of weakness. Table 4-17 
presents the result of the observations. As table 4-17 shows, all assessors received scores of 85 percent or 
better while being observed. 
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Number of Check Marks 



Exhibit 4-2 1 . Proportion Correct Chart: School year 2001-02 



Observation Scoring 
Proportion Correct Chart 

Total Possible Points 





69 


70 


72 


73 


74 


76 


77 


78 


79 


80 


81 


82 


83 


84 


85 


86 


87 


88 


89 


90 


92 


1 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


0.99 


2_ 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


0.98 


3_ 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.96 


0.97 


0.97 


0.97 


0.97 


0.97 


0.97 


4_ 


0.94 


0.94 


0.94 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.95 


0.96 


0.96 


0.96 


5_ 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.94 


0.95 


6_ 


0.91 


0.91 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


0.93 


7 


0.90 


0.90 


0.90 


0.90 


0.91 


0.91 


0.91 


0.91 


0.91 


0.91 


0.91 


0.91 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


0.92 


8_ 


0.88 


0.89 


0.89 


0.89 


0.89 


0.89 


0.90 


0.90 


0.90 


0.90 


0.90 


0.90 


0.90 


0.90 


0.91 


0.91 


0.91 


0.91 


0.91 


0.91 


0.91 


9_ 


0.87 


0.87 


0.88 


0.88 


0.88 


0.88 


0.88 


0.88 


0.89 


0.89 


0.89 


0.89 


0.89 


0.89 


0.89 


0.90 


0.90 


0.90 


0.90 


0.90 


0.90 


10_ 


0.86 


0.86 


0.86 


0.86 


0.86 


0.87 


0.87 


0.87 


0.87 


0.88 


0.88 


0.88 


0.88 


0.88 


0.88 


0.88 


0.89 


0.89 


0.89 


0.89 


0.89 


11 


0.84 


0.84 


0.85 


0.85 


0.85 


0.86 


0.86 


0.86 


0.86 


0.86 


0.86 


0.87 


0.87 


0.87 


0.87 


0.87 


0.87 


0.88 


0.88 


0.88 


0.88 


12_ 


0.83 


0.83 


0.83 


0.84 


0.84 


0.84 


0.84 


0.85 


0.85 


0.85 


0.85 


0.85 


0.86 


0.86 


0.86 


0.86 


0.86 


0.86 


0.87 


0.87 


0.87 


13_ 


0.81 


0.81 


0.82 


0.82 


0.82 


0.83 


0.83 


0.83 


0.84 


0.84 


0.84 


0.84 


0.84 


0.85 


0.85 


0.85 


0.85 


0.85 


0.85 


0.86 


0.86 


14_ 


0.80 


0.80 


0.81 


0.81 


0.81 


0.82 


0.82 


0.82 


0.82 


0.83 


0.83 


0.83 


0.83 


0.83 


0.84 


0.84 


0.84 


0.84 


0.84 


0.84 


0.85 


15_ 


0.78 


0.79 


0.79 


0.79 


0.80 


0.80 


0.81 


0.81 


0.81 


0.81 


0.81 


0.82 


0.82 


0.82 


0.82 


0.83 


0.83 


0.83 


0.83 


0.83 


0.84 


16_ 


0.77 


0.77 


0.78 


0.78 


0.78 


0.79 


0.79 


0.79 


0.80 


0.80 


0.80 


0.80 


0.81 


0.81 


0.81 


0.81 


0.82 


0.82 


0.82 


0.82 


0.83 


17 


0.75 


0.76 


0.76 


0.77 


0.77 


0.78 


0.78 


0.78 


0.78 


0.79 


0.79 


0.79 


0.80 


0.80 


0.80 


0.80 


0.80 


0.81 


0.81 


0.81 


0.82 


18_ 


0.74 


0.74 


0.75 


0.75 


0.76 


0.76 


0.77 


0.77 


0.77 


0.78 


0.78 


0.78 


0.78 


0.79 


0.79 


0.79 


0.79 


0.80 


0.80 


0.80 


0.80 


19_ 


0.72 


0.73 


0.74 


0.74 


0.74 


0.75 


0.75 


0.76 


0.76 


0.76 


0.77 


0.77 


0.77 


0.77 


0.78 


0.78 


0.78 


0.78 


0.79 


0.79 


0.79 


20 _ 


0.71 


0.71 


0.72 


0.73 


0.73 


0.74 


0.74 


0.74 


0.75 


0.75 


0.75 


0.76 


0.76 


0.76 


0.76 


0.77 


0.77 


0.77 


0.78 


0.78 


0.78 


21 


0.70 


0.70 


0.71 


0.71 


0.72 


0.72 


0.73 


0.73 


0.73 


0.74 


0.74 


0.74 


0.75 


0.75 


0.75 


0.76 


0.76 


0.76 


0.76 


0.77 


0.77 


22 _ 


0.68 


0.69 


0.69 


0.70 


0.70 


0.71 


0.71 


0.72 


0.72 


0.73 


0.73 


0.73 


0.73 


0.74 


0.74 


0.74 


0.75 


0.75 


0.75 


0.76 


0.76 


23 _ 


0.67 


0.67 


0.68 


0.68 


0.69 


0.70 


0.70 


0.71 


0.71 


0.71 


0.72 


0.72 


0.72 


0.73 


0.73 


0.73 


0.74 


0.74 


0.74 


0.74 


0.75 


24 _ 


0.65 


0.66 


0.67 


0.67 


0.68 


0.68 


0.69 


0.69 


0.70 


0.70 


0.70 


0.71 


0.71 


0.71 


0.72 


0.72 


0.72 


0.73 


0.73 


0.73 


0.74 


25 _ 


0.64 


0.64 


0.65 


0.66 


0.66 


0.67 


0.68 


0.68 


0.68 


0.69 


0.69 


0.70 


0.70 


0.70 


0.71 


0.71 


0.71 


0.72 


0.72 


0.72 


0.73 


26 _ 


0.62 


0.63 


0.64 


0.64 


0.65 


0.66 


0.66 


0.67 


0.67 


0.68 


0.68 


0.68 


0.69 


0.69 


0.69 


0.70 


0.70 


0.70 


0.71 


0.71 


0.72 


27 


0.61 


0.61 


0.63 


0.63 


0.64 


0.64 


0.65 


0.65 


0.66 


0.66 


0.67 


0.67 


0.67 


0.68 


0.68 


0.69 


0.69 


0.69 


0.70 


0.70 


0.71 


28 _ 


0.59 


0.60 


0.61 


0.62 


0.62 


0.63 


0.64 


0.64 


0.65 


0.65 


0.65 


0.66 


0.66 


0.67 


0.67 


0.67 


0.68 


0.68 


0.69 


0.69 


0.70 


29 _ 


0.58 


0.59 


0.60 


0.60 


0.61 


0.62 


0.62 


0.63 


0.63 


0.64 


0.64 


0.65 


0.65 


0.65 


0.66 


0.66 


0.67 


0.67 


0.67 


0.68 


0.68 


30 


0.57 


0.57 


0.58 


0.59 


0.59 


0.61 


0.61 


0.62 


0.62 


0.63 


0.63 


0.63 


0.64 


0.64 


0.65 


0.65 


0.66 


0.66 


0.66 


0.67 


0.67 



NOTE: Proportion Correct = (Total Possible Points - Number of Check Marks)/Total Possible Points 

EXAMPLE: If an Assessor received 8 check marks out of a total of 86 possible points, the proportion correct is 0.91 . This Assessor passed. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, school year 
2001 - 02 . 





Exhibit 4-22. Example of Proportion Correct Chart: School year 2001-02 



Step 6: Use Proportion Correct Chart to determine the proportion correct and write that proportion in 


this box: 


.98 





SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Table 4-17. Results of the child assessments observations, spring-third grade: 
School year 2001-02 



Observations* 


Number 


Percent 


Total 


468 


100 


Score on certification form 


85 percent or above 


468 


100 


70-84 percent 


0 


0 


Below 70 percent 


0 


0 



' Two hundred and fifty-eight assessors were initially observed. Of these 200 received a second observation visit and 10 
received a third observation visit. No assessor scored below 85 percent on any of the observation visits. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998—99 third grade data collection, school year 2001-02. 



4.6.1.2 Assessor Interrater Reliability 

As part of the child assessment observation described above, field supervisors completed an 
Assessment Certification Form for each observation they conducted. An important element of this form 
was the “validation items.” With the exception of the reading routing test, all of the assessments included 
at least one item that both the observer and the assessor scored. These items had open-ended responses 
that called for interpretation on the part of the assessor to determine whether a child’s response was 
correct. By comparing the extent to which assessors and observers agreed on scoring these validation 
items, a measure of interrater reliability was obtained. Interrater reliability provided a measure of the 
accuracy of the assessor’s scoring compared with the standard, the observer’s. Supervisors had been 
certified at training by experienced Westat staff. New supervisors were first observed by field managers 
before observing any assessments. 

Table 4-18 contains the results of these comparisons. The overall interrater reliability was 
very high throughout all the forms. The average percent agreement between assessors and observers was 
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7.5 percent for reading, 99.4 percent for mathematics, and 98.2 percent for science. The lowest percent 
agreement was 96.9 percent (for Reading Red and Yellow and 96.8 percent for Science Yellow). 



Table 4-18. Interrater reliability on child assessment validation items by domain level, 
spring-third grade: School year 2001-02 



Category 


Number of 
observations 


Number of 
validation items 


Average percent 
agreement: 
assessors and observers 


Reading 


461 


13 


97.5 


Routing 


t 


0 


t 


Red 


118 


3 


96.9 


Yellow 


261 


5 


96.9 


Blue 


82 


5 


98.7 


Mathematics 


464 


7 


99.4 


Routing 


468 


1 


99.6 


Red 


135 


1 


99.3 


Yellow 


186 


2 


99.7 


Blue 


143 


3 


99.1 


Science 


451 


12 


98.2 


Routing 


458 


3 


98.0 


Red 


140 


1 


100.0 


Yellow 


226 


4 


96.8 


Blue 


85 


4 


97.9 



t Not applicable. No reading routing items were selected for validation. 

NOTE: Percent agreement for specific domain levels (e.g., Reading Red) was calculated as follows: number of validation items 
observed in which observer agreed with the assessor divided by the total number of validation items observed. Percent agreement for 
domains (e.g., Science) is an average of the percent agreements for each level within that domain. The number of observations at the 
domain level (Reading, Mathematics, Science) is the sum of the observations at each specific domain level (Red, Yellow, Blue) to 
reflect the typical number of children assessed within a given domain. Due to missing data, the sum of observations for the specific 
domain levels do not sum to the number of observations at the routing level. The missing data occurred because not every child was 
able to answer all the questions in the assessment. All children, however, were able to complete the routing forms for the domains. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 
Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 



4.6.2 Validation of Parent Interviews 

Approximately 10 percent of the completed parent interviews were called back by a field 
supervisors (i.e., validated). The first parent interview completed by an assessor was always validated. 
Over the course of the field period, a running count of an assessor’s completed parent interviews was 
maintained, and each tenth completed parent interview was selected for validation, thus ensuring that 1 0 
percent of each assessor’s cases were selected for validation. The parent validation was approximately 5 
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minutes long and was conducted by telephone. No in-person cases were validated because such 
validations would have to be done in person. 



Field supervisors used a standardized parent validation script, the Parent Interview 
Validation Form (see appendix H2: Parent Interview Validation Form), to make validation calls to 
parents. The validation script included verification of the child’s name, date of birth, and sex, and 8 to 
10 questions from the parent interview. Parent Validation Interview Forms were generated at the end of 
May and calls to the parents began in June. By the end of July, a full 10 percent (1,324) of the total 
completed parent interviews had been validated. Validation results were as follows: (1) “No changes” 
meaning responses to the original interview and the validation interview were identical; (2) “Minor 
changes” meaning there was a minor discrepancy (e.g., the ZIP code was different) between the responses 
to the original interview and the validation interview; and (3) “Major changes” meaning there was a 
discrepancy between the response to the original interview and the response in the validation interview 
(e.g., the school breakfast program was reported differently between the two contacts). One interviewer’s 
parent validation determined that the interview was falsified. The date when the problem occurred was 
identified and all cases the interviewer completed after that date was validated. Random interviews before 
the date when the problem occurred for that interviewer were also selected for validation. In total, 20 
cases were validated. The five cases that were found to be falsified were re-set to pending, sent out to the 
field, and completed by another interviewer. The interviewer who falsified data was released immediately 
from the project. Table 4-19 presents the results of the parent validations. 

Table 4-19. Results of parent interview validations: School year 2001-02 



May 


June 


July 


Parent validation status 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Completed parent interviews 


9,044 


100.0 


11,590 


100.0 


13,220 


100.0 


Validation cases generated 


1,150 


12.7 


1,473 


12.7 


1,583 


12.0 


Validation cases receipted 


0 


0.0 


972 


66.0 


1,324 


83.6 


No changes 


0 


0.0 


611 


62.9 


836 


63.1 


Minor changes 


0 


0.0 


209 


21.5 


273 


20.6 


Major changes 


0 


0.0 


141 


14.5 


201 


15.2 


Other (specify) 


0 


0.0 


11 


1.1 


14 


1.1 


Cases pending 


1,150 


100.0 


501 


34.0 


259 


16.4 



NOTE: The pending cases at the end of July were cases for which the respondent could never be reached to verity the completion of the interview 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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4.6.3 



Validations of School Visits 



To ensure that assessments proeeeded smoothly, a validation call was completed with the 
school principal in at least two of each supervisor’s assigned schools in the spring-third grade data 
collection. Field managers conducted the school validations. 

The first school that each team completed was called to ascertain how well the 
preassessment and assessment activities went. If the feedback from the school was positive, the fifth 
school that each team completed was called. If any problems were indicated in the first validation call, 
immediate action was taken with the field supervisor. The validation feedback was discussed with the 
supervisor and remedial action was taken, including in-person observation of the supervisor’s next school, 
if necessary. 



Field managers used a standardized telephone script, the School Validation Script (see 
appendix H3: School Validation Form), to call the school principals. The script covered the following 
topics: 



■ An overall rating of how the assessments went; 

■ Feedback the principal had received about the study from children and teachers; 

■ Suggestions for improving procedures and making it easier for a school to participate; 
and 

■ General comments and suggestions. 

By the end of the field period in June, field managers had called 155 original sample schools 
(100 percent of the schools to be validated) to ascertain whether the conduct of the child assessments in 
the school was satisfactory. The most common feedback from school administrators was on the burden of 
the hard-copy questionnaires. About 15 percent of school administrators reported that the school and 
teacher questionnaires were a “real burden’’ or “too long’’ and that there was too much redundancy 
between the school and teacher questionnaires. School administrators recommended shortening the 
questionnaires considerably to make it less stressful for the staff and make it easier for teachers to 
complete questionnaires. Some school administrators suggested that the study provide the data that was 
collected from them in previous years so that they could update the items. Although teachers were not 
included in the validations, some of them sent unsolicited e-mails, letters, and telephone calls indicating 
that they were disappointed with the remuneration, thinking it was insufficient compensation for the 
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amount of work they were asked to do. Almost three quarters (72 percent) of the teachers in original 
sample schools were linked to three or fewer children; more than half (58 percent) were linked to only 
one or two children. Given that the remuneration was $7 per child, some teachers perceived the incentive 
as an insult. Table 4-20 presents the results of the school validation calls. 



Table 4-20. Results of school validation calls: School year 2001-02 



Slrhnnl valiHatinn 


May 




June 






Number 


Percent 


Number 


Percent 


Total validation cases generated 


155 


100.0 


155 


100.0 


Validation cases completed 


113 


72.9 


150 


96.8 


Overall rating 










Very satisfactory 


108 


95.6 


145 


96.7 


Satisfactory 


4 


3.5 


4 


2.7 


Unsatisfactory 


0 


0.0 


0 


0.0 


Very unsatisfactory 


1 


0.9 


1 


0.7 


Validation cases not completed 


42 


17.1 


5 


3.2 



NOTE: Detail may not sum to totals due to rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class 
of 1998-99 third grade data collection, school year 2001-02. 
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5. DATA PREPARATION AND EDITING 



As described in chapter 4, two types of data collection instruments were used for the Early 
Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) data collection in the spring- 
third grade: computer-assisted and self-administered paper forms (hard copy). The data preparation 
approach differed with the mode of data collection. The direct child assessments and parent interview 
were conducted using computer-assisted interviewing (CAI) techniques. Editing specifications were built 
into the computer programs used by assessors to collect these data. The teacher, school administrator, and 
student records abstract forms were self-administered. When the field supervisors and respondents 
returned these forms, coders recorded the receipt of these forms into a project-specific forms tracking 
system. Prior to mailing the questionnaires and forms to Westat, the field supervisors reviewed them to 
ensure that critical items had been completed (see appendix E6 for a list of the critical items). When 
forms and questionnaires were received at Westat, coders reviewed them to ensure data readability for 
transfer into an electronic format and for the completion of critical items. The visual review included 
changing (upending) any “Other, specify” responses that actually fit within the available response 
categories of the question. For example, if the parent said “skiing” in response to the question on the types 
of exercise or physical activity the child participated in, this answer was upcoded to the existing category, 
“individual sports.” (See appendix I: “Other, specify” Items and Coding Instructions” for more 
information on the coding of “other” responses.) There were some items for which upcoding was 
conducted after the data were keyed due to the large volume of “Other” responses. Once they finished this 
review, the coders sent the instruments to data entry to be manually transferred to an electronic format 
and reviewed for range and logic consistency. The following sections describe the data preparation 
activities for both modes of data collection in more detail. 



5.1 Coding and Editing Specifications for Computer-Assisted Interviews (CAI) 

The very nature of designing a computer-assisted interview requires decisions about edit 
specifications to be made at the development stage. Both acceptable ranges and logic consistency checks 
were preprogrammed into the electronic questionnaires (see appendix R: Range Specifications for Each 
Questionnaire and appendix L: Parent Interview Household Roster Data Edits for ranges and logic 
consistency checks). The next few sections describe the coding and editing of the data that were 
conducted during and after CAI parent interview. 
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5.1.1 



Range Specifications 



Within the CAI parent interview instruments, respondent answers were subjected to both 
“hard” and “soft” range edits during the interviewing process. A “soft range” is one that represents the 
reasonable expected range of values but does not include all possible values. Responses outside the soft 
range were confirmed with the respondent and entered a second time. For example, the number of hours 
each week a child attended a day care center on a regular basis had a soft range of 1 to 50. A value outside 
this range could be entered and confirmed as correct by the interviewer as long as it was within the hard 
range of values (1 to 70). 

“Flard ranges” have a finite set of parameters for the values that can be entered into the 
computer, for example, “0-5 times” for the number of times the child, in the previous 5 days, ate a 
breakfast that was not school provided. Out-of-range values for closed-ended questions were not 
accepted. If the respondent insisted that a response outside the hard range was correct, the interviewer 
could enter the response in a comments data file. Data preparation and project staff reviewed these 
comments. Out-of-range values were accepted and entered into the data file if the comments supported 
the response. 



The child assessments did not employ hard and soft ranges. Children’s answers were 
recorded verbatim. 



5.1.2 Logic Consistency Checks (Logical Edits) 

Logic consistency checks, or logical edits, examined the relationship between responses to 
ensure that they did not conflict with one another or that the response to one item did not make the 
response to another item unlikely. For example, in the household roster, one could not be recorded as a 
mother and male. When a logical error such as this occurred during an interviewing session, the 
interviewer was presented with a message requesting verification of the last response and a resolution of 
the discrepancy. In some instances, if the verified response still resulted in a logical error, the interviewer 
recorded the problem either in a comment field or on a problem report. Consistency checks were not 
applicable to the child assessments. 
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5.1.3 



Coding 



Additional coding was required for some of the items collected in the CAI instruments by 
data preparation and project staff after an interview was completed. These items included “Other, specify” 
text responses, occupation, and race/ethnicity. Interviewers entered verbatim responses to these items. 
Data preparation staff were trained to code these data using coding manuals designed by Westat and the 
National Center for Education Statistics (NCES) to support the coding process. This section describes the 
coding activities for the CAI instruments. 



5.1.3. 1 Review of “Other, Specify” Items 

There were sixteen “Other, specify” open-ended responses in the parent interview (see 
appendix I for the complete set of “Other, specify” items and coding instructions). All of these items were 
reviewed to determine if they should be coded into one of the existing response categories. During data 
collection, when a respondent selected an “other” response in the parent interview, the interviewer entered 
the text into a “specify” overlay that appeared on the screen. The data preparation staff reviewed these 
text “specify” responses and, where appropriate, coded them into one of the existing response categories. 
If a response did not fit into one of the existing categories, it remained in “other.” If there were numerous 
responses that were essentially the same, a new category was added. The addition of new categories was 
not needed during the third grade data collection, but it did occur in earlier rounds (see appendix J: Added 
Response Codes From Previous Rounds). 

The parent “Other, specify” coding system was revised from previous rounds of data 
collection and in production testing in April 2002. The revisions consisted of adding new “Other, specify” 
items that had not been part of the previous rounds. Parent “Other, specify” coding was completed in 
batches of cases during June and July 2002; the first batch of 3,333 text strings was extracted in June, 
2002, and the last batch of 2,320 text strings was extracted in July. A total of 5,653 “Other, specify” text 
strings were processed through the parent “Other, specify” coding system. All possible opcodes were 
applied to the 4,204 cases that had at least one “Other, specify” text string. As noted above, whenever 
appropriate, responses were upcoded to existing categories. There were no “Other, specify” items in the 
child assessments. Table 5-1 presents the number of text strings for each “Other, specify” item. 
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Table 5-1. Number of text strings by “Other, specify” item, third grade parent interview: 
School year 2001-02 



Other, specify item description 


Item 


Number of strings 


Total 




5,653 


Behavior Diagnosis 


CHQ.337 


18 


Birth Place 


FSQ.240 


1 


Breakfast Place 


HEQ.117 


87 


Child Support Agreement 


NRQ.261 


113 


Center Pay Unit 


CCQ.393 


18 


Discipline 


DWQ.no 


662 


Doing East Week 


EMQ.080 


335 


Emotional Diagnosis 


CHQ.365 


207 


Exercise Types 


CHQ.726 


653 


Get to School 


HEQ.125 


513 


Housing 


PAQ.140 


190 


Interview Eanguage 


CMQ.690 


41 


Job Find Effort 


EMQ.070 


116 


Mental Diagnosis 


CHQ.060 


198 


Nonrelative 


FSQ.181 


192 


Nonrelative Pay Unit 


CCQ.225 


1 


Other, specify Stamps 


WPQ.130 


10 


Other, specify TANF 


WPQ.106 


6 


Race 


FSQ.198 


158 


Reason Eeft Household 


FSQ.015 


737 


Relative Pay Unit 


CCQ.128 


2 


Tutor Subject 


HEQ.107 


108 


What Act Diagnosis 


CHQ.120 


9 


Why Call School 


PIQ.018 


1,278 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, 



Kindergarten Class of 1998-99 third grade data collection, school year 2001-02. 



5.1.3.2 Parent Occupation Coding 

As in the base year and first grade data collections, occupations were coded using the 
Manual for Coding Industries and Occupations (U.S. Department of Education, 1999). This coding 
manual was created for the Adult Education Survey of the National Household Education Surveys 
Program (NHES) and used an aggregated version of industry and occupation codes (see appendix K: 
Industry and Occupation Code Descriptions). The industry and occupation codes used by NHES were 
originally developed for the 1989-90 National Postsecondary Student Aid Study (NPSAS) and contained 
one to four digits. Analysis of the NPSAS categories revealed that some categories had very small 
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numbers of cases and similar categories had similar participation rates, suggesting that the separate codes 
could be collapsed without significant loss of information. The NHES industry and occupation code 
categories use a two-digit code, the highest level of aggregation, to have sufficient numbers of cases to 
support analysis without collapsing categories. There are 13 industry codes and 22 occupation codes in 
the NHES coding scheme. If an industry or occupation could not be coded using this manual, the Index of 
Industries and Occupations — 1980 (U.S. Department of Commerce, 1982) and Standard Occupational 
Classification Manual — 1980 (U.S. Department of Commerce, 1980) were used. Both of these manuals 
use an expanded coding system and at the same time are directly related to the much more condensed 
NHES coding scheme. These manuals were used for reference in cases where the NHES manual did not 
adequately cover a particular situation. (See appendix K for an expanded description of the industry and 
occupation codes.) 

Occupation coding began with an autocoding procedure using a computer string match 
program developed for the NHES. The program searched the responses for strings of text for each 
record/case and assigned an appropriate code. A little over half the cases were autocoded (50.4 percent). 
Cases that could not be coded using the autocoding system were coded manually using a customized 
coding utility program designed for coding occupations. The customized coding utility program brought 
up each case for coders to assign the most appropriate codes. In addition to the text strings, other 
information, such as main duties, highest level of education, and name of the employer, was available for 
the coders. The coders used this information to ensure that the occupation code assigned to each case was 
appropriate. 



After the cases were coded (either manually or via autocoding), they were reviewed and 
verified. One hundred percent of the cases were verified. Verification of coding is an important tool for 
ensuring quality control and extending coder training. As a verification step, a second coder 
independently assigned codes (i.e., double-blind coding) to industry and occupation cases that had been 
initially coded either by the autocoding system or manually by a coder. A coding supervisor adjudicated 
disagreements between the initial code and the verification code. In the early stages, 1 00 percent of each 
coder’s work was reviewed. Once the coder’s error rate had dropped to 1 percent or less, 10 percent of the 
coder’s work was reviewed. Almost 20 percent (19.9 percent) of the cases that were autocoded required 
adjudication because the verifier disagreed with the autocoding. About the same percent (21.2 percent) of 
the manually coded cases required adjudication because the manual coder and the verifier disagreed. 
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Table 5-2 summarizes the results of the coding and verification process for occupation 
coding. In the table, manually coded indicates that occupation was initially coded by a coder as opposed 
to using the autocoding system. Discrepancies are the count of disagreements between the autocoder and 
the verifier or between the manual coder and the verifier: the discrepant cases required adjudication. The 
percentage of times in which the coding supervisor disagreed with the coder’s (or the autocoding 
system’s) initial coding is referred to as the coder error rate. The percentage of times in which the coding 
supervisor disagreed with the verifier’s coding is referred to as the verifier error rate. The denominator 
used in calculating these error rates is the number of cases verified. The error rate for manually coded 
cases was essentially the same for coders and verifiers (11.7 percent and 12.0 percent). The autocoded 
cases had a slightly higher coder error rate, 13.7 percent, compared to the manually coded rate, 11.7 
percent. 



Table 5-2. Number and percent of occupations coded, by coding status: School year 2001-02 



Coding Status 


Number coded 


Percent 


Total 


9,392 


100.0 


Coded 


9,392 


100.0 


Autocoded 


4,732 


50.4 


Manually coded 


4,660 


49.6 


Verified 


9,392 


100.0 


Verified from autocoding 


4,732 


50.4 


Verified from manual coding 


4,660 


49.6 


Discrepancies between coding and verifying 


1,927* 


100.0 


Adjudicated from autocoding 


941 


48.5 


Adjudicated from manual coding 


986 


51.2 


Autocoding system or manual coder wrong 


1,192 


100.0 


Autocoded 


648 


54.4 


Manually coded 


544 


45.6 


Verifier wrong 


960 


100.0 


Autocoded 


401 


41.8 


Manually coded 


559 


58.2 



NOTE: Occupation was collected for up to two key persons in spring-third grade. It was only collected for persons who had a different job from 
spring-first grade or who had not been employed in the previous round, but were employed in spring-third grade. For everyone else, the spring- 
first grade data were carried forward. The numbers in this table represent occupational text strings that were coded into appropriate occupation 
categories and applied to the appropriate key person (e.g., mother or father). 

* Total discrepancies do not equal the sum of coder wrong and verifier wrong because sometimes both coder and verifier were deemed to be 
wrong. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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5.1.4 



Editing the Household Roster in the Parent Interview 



The parent data edit system was modified from spring-first grade in April and May 2002 to 
reflect changes to the third grade parent interview. The parent interview data were edited in batches as the 
interviews were completed (see table 5-3). This was done to make the process more efficient. The first 
batch consisted of all cases received from the beginning of the round through June 13, 2002. The second 
batch consisted of cases completed between June 13 and July 9. The last batch consisted of all the 
remaining cases through the end of the round. 



Table 5-3. Data editing of the parent interview household roster, third grade data collection: School 
year 2001-02 



Batch 

number 


Data 

extraction date 


Number of 
households 
extracted 


Percent of 
total households 


Number of 
households 
failing edits 


Percent of 
households 
extracted 


Total 


t 


13,360 


100 


406 


3.0 


1 


6/13/02 


8,112 


60.7 


246 


3.0 


2 


7/9/02 


3,926 


29.4 


107 


2.8 


3 


8/6/03 


1,322 


9.9 


53 


4.0 



t Not applicable. 

NOTE: The data in this table are household-level data, not case-level data (that is, not child-level). Detail may not sum to totals due to rounding. 
SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



The first step in the editing process was to extract the household roster data and run the data 
edits. The second step was to apply the programmatic updates to the cases failing the edits to correct any 
errors programmatically. The third step was for an expert reviewer to manually review the cases, conduct 
as-needed discussions with NCES for resolution, and resolve and correct data errors. Four cases required 
a discussion with NCES. Only one case was lost from the data file after the expert review and discussion 
with NCES because the respondent was a nonresident father and thus was not eligible to participate. 

Several tests were run on the household roster to look for missing or inaccurate information. 
There were essentially three general types of roster tests performed to determine which cases required 
editing (see appendix E: Parent Interview Household Roster Data Edits). First, the relationship of an 
individual to the focal child was compared to the individual’s listed age and sex. Problems found were 
corrected on the basis of data from prior data collections wherever possible. Second, households with 
more than one mother or more than one father were scrutinized for errors. While it was possible to have 
more than one mother in a household — for example, a household could contain one biological and one 
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foster mother of the focal child — such cases warranted closer inspection. Corrections were made 
whenever clear errors and a clear resolution existed. Last, the relationship of an individual to both the 
focal child and the respondent was examined, as there were cases in which the relationship of an 
individual to the focal child conflicted with his status as the spouse/partner of the reference person. For 
example, in a household containing a child’s grandparents but not his or her parents, the grandmother may 
be designated the “mother” figure, and the grandfather thus becomes the “father” (for the purposes of 
some questions in the interview) by virtue of his marriage to the grandmother. These cases were 
examined but left unchanged. Both the original — and correct (grandfather) — relationship data and the 
new “parent-figure” designation (father) that had been constructed were kept. 

Updates to the household roster were required when one or more of the edit checks described 
above failed. The data for the case would be inspected, resolved, and, if necessary, corrected. There were 
644 parent interview cases requiring edits in third grade (identified by P5EDIT=1). There were 164 cases 
in which the interviewer noted in FSQ that they had entered a person incorrectly in the household roster. 
These cases can be identified by the flag “P5ERRFLG” on the final data files. These “error” cases may or 
may not have also been edit cases, depending on whether or not the error triggered one of the edit checks. 



5.2 Coding and Editing Specifications for Hard-Copy Questionnaires 

5.2.1 Receipt Control 

In order to monitor the more than 50,000 documents that were to be received in the third 
grade year, the project-specific receipt and document control system developed in the base year was used, 
with some modifications. The electronic receipt and document control system was initially loaded or 
filled with identifying information, such as identification numbers for schools, teachers, and children; the 
identification numbers linking teachers and children; and the questionnaires that were expected from each 
school and teacher for each cooperating school in the sample. As data were collected in the field, field 
supervisors completed transmittal forms for each school to indicate which questionnaires were being 
mailed to the home office. Teachers did not complete transmittal forms. They mailed their questionnaires 
to Westat in the Federal Express mailers that were provided to them. 

Once data collection started, receipt control clerks reviewed the questionnaires sent in from 
the field for accuracy and completeness. The identification number on each form was matched against the 
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identification numbers in the tracking system to verify that the appropriate number of forms for each 
school was returned. Questionnaires that matched identification numbers in the tracking system were 
receipted in the system and assigned a batch number. All receipted questionnaires matched identification 
numbers in the tracking system. Each questionnaire type had a different batch number so that 
questionnaires of the same type would be processed together. Processing data in batches was more 
efficient because it allowed staff to process in volume one type of questionnaire rather than switching 
from one type to another. Questionnaires with the same batch numbers were then compiled into batches 
of 25 instruments. The batch sheets with all the questionnaire identification numbers were printed from 
the system; the questionnaires in the batches were verified against the batch sheets. Verified batches were 
then sent for data entry. Once the batches had completed data entry, that is, all the questionnaires in a 
given batch had been keyed, the batches, accompanied by the electronic data, were returned to the data 
preparation department. The electronic data were loaded into the editing system and edited. At each point 
in the process, a flag was set in the receipt and document control system, which indicated the status of the 
instrument and the batch to which it was assigned. These statuses were the following: 

■ Batches created; 

■ Batches printed; 

■ Batches verified; 

■ Batches sent to keypunch; 

■ Batches logged from keypunch; 

■ Cases loaded to BES (editing system); 

■ Cases cleaned (in edit system); and 

■ Cases pending (in edit system). 



5.2.2 Scan Edit Procedures 

Prior to receipting returned questionnaires, trained clerks scanned each instrument for 
missing critical items (see appendix Ml: Receipt and Scan Edit Procedures Manual). Critical items were 
identified for each hard-copy questionnaire, except for the facilities checklist. Questionnaires with 
incomplete or missing data for critical items were not considered complete and were processed for return 
to the field. Clerks completed scan edit sheets listing the missing critical items. The questionnaires were 
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receipted in the system as “needs data retrieval” and were forwarded to the data preparation department 
for data entry, coding, and editing. Using the scan edit sheets, clerks identified the missing or incomplete 
items on data retrieval forms that were then sent to the appropriate field supervisor (see appendix M2: 
Scan Edit Rules for copies of the forms used to record missing or incomplete critical items). The 
supervisor was instructed to contact the school to try to obtain the missing information. See 
section 4.5.5. 1 for further discussion of the data retrieval data collection procedures. 



Questionnaires that were scanned and needed no data retrieval were logged into the receipt 
and document control system as “complete, no data retrieval.” Once forms were logged in, the data were 
first keypunched into electronic format and than coded and edited. Questionnaires that contained no data 
due to refusal by the respondents were logged into the receipt and document control system as “refusal.” 
Table 5-4 presents data on the number of questionnaires receipted by week. 



Table 5-4. Number of questionnaires receipted by week, third grade data collection: School year 2001-02 



Week 


Date 


Number receipted 


Cumulative number receipted 


Percent of total receipted 


Total 




40,319 


40,319 


100 


1 


4/1/02 


2,172 


2,172 




2 


4/8/02 


2,486 


4,658 


11.6 


3 


4/15/02 


3,707 


8,365 


20.8 


4 


4/22/02 


2,957 


11,322 


28.2 


5 


4/29/02 


1,176 


12,498 


31.1 


6 


5/1/02 


1,118 


13,616 


33.9 


7 


5/6/02 


2,342 


15,958 


39.8 


8 


5/13/02 


2,122 


18,080 


45 


9 


5/20/02 


2,302 


20,382 


50.7 


10 


5/28/02 


2,596 


22,978 


57.2 


11 


6/3/02 


4,893 


27,871 


69.4 


12 


6/10/02 


3,974 


31,845 


79.3 


13 


6/17/02 


3,156 


35,001 


87.2 


14 


6/24/02 


3,007 


38,008 


94.7 


15 


7/1/02 


1,399 


39,407 


98.2 


16 


7/8/02 


404 


39,811 


99.2 


17 


7/15/02 


249 


40,060 


99.8 


18 


7/22/02 


259 


40,319 


100 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



The following sections describe the data entry, coding, and editing processes for hard-copy 
questionnaires. 
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5.2.3 



Data Entry 



Data entry consisted of two steps: (1) entering the data and (2) verifying that the data had 
been entered accurately. Westat data entry staff keyed the forms in each batch. A set of keying rules was 
established for each questionnaire that the data entry staff followed (see appendix N: Keying Instructions 
For Questionnaires). To verify the accuracy of the data entry, more senior data entry operators then 
rekeyed all the data. The results of the two data entry sessions were compared and differences identified. 
When differences were found, the hard-copy form was pulled and examined to determine what 
corrections, if any, had to be made to the keyed data. These corrections were rekeyed, resulting in an 
accuracy rate exceeding 99 percent. The verified batches were then transmitted electronically to Westat’s 
computer system for data editing. 



5.2.4 Coding 

The hard-copy questionnaires required a quick visual review of particular questions in each 
questionnaire, coding of race/ethnicity for teachers, and review of “Other, specify” text responses. The 
quick visual review was to ensure that the questionnaire values accurately reflected existing categories, 
were complete and consistent across variables, and that the numbers were converted to the appropriate 
unit of measurement prior to converting data to an electronic format. Once the hard-copy questionnaires 
had been visually reviewed, they were coded. The coding staff was trained on the procedures and had 
manuals to support the coding process (see appendix O: Coding Instructions and Items To Be Coded). 
Senior coders verified coding. The verification rate was set at 1 00 percent for each coder until accuracy of 
less than 1 percent error rate was established. After that point, work was reviewed at a rate of 1 0 percent. 

The “Other, specify” text responses were reviewed by the data editing staff and, where 
appropriate, upcoded into one of the existing response categories. The specify responses that remained 
after upending were reviewed to evaluate whether the addition of any new response categories would be 
appropriate. There was no need for the addition of new response categories in third grade (see appendix P: 
Codes From Previous Rounds of Data Collection). 
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5.2.5 



Data Editing Management Process 



The management of the data editing process involved the creation of several data fdes, 
including the Collection Database, Holding Database, Editing Database, and Delivery Database. 
Exhibit 5-1 provides a diagram of the process described below. 



5.2.5.1 Collection Database 

This database contained the keyed records for hard-copy questionnaires. One Collection 
Database was created for each instrument and as additional data were keyed, the cases were added to the 
database. The Collection Databases were Blaise databases. The ASCII file resulting from the key entry 
process was converted to Blaise data in the Collection Database so that they could be merged with the 
parent interview data and so that they could undergo additional data review (see section 5.2.6) 

Records in the Collection Databases were assigned status codes reflecting their current 
status. All new records were assigned a status of CollectionNew. When cases were copied to the Holding 
Database, the status was updated to CollectionCopied. The data in the Collection Database were retained 
in their original form, that is, they were not modified based upon later steps. 



5.2.5.2 Holding Database 

Data were copied from the Collection Database to the Holding Database for the editing 
process. The Holding Database for each instrument was also a Blaise database. The copied cases were 
assigned a status code of New. Cases that had already been involved in a prior editing cycle and had been 
returned to the Holding Database were assigned a status of CheckEdit or KeepAsIs. 
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Exhibit 5-1. ECLS-K hard-copy data editing, third grade data collection: School year 2001-02 



From Field 



Copy as is. Set Collection Status to 
CollectionNew 



Collection 






Check New and CheckEdit cases, 
set Fielding status as follows: 

- if case is DIRTY 

- if status was New, set status to 
DirtyNew. 

- if status was CheckEdit, set 
status to DirtyEdit. 

- if case is CLEAN 

- if status was New, set status to 
CleanNew. 

- if status was CheckEdit, set 
status to CleanEdit. 



Other assorted edits and 
transpose issues. 

AddIv other soecifv uocodes. 
Floldina status would be set to 
InDelivery. 



Copy to Fielding and set Fielding status to New. Set 
Collection status to CollectionCopied. Apply code-all- 
that-apply recoding and yes/no recoding. Extract 
Other Specify text. Convert all blanks to -9, and all 
don't knows and refused to -8 and -7 as appropriate. 



Error Files 




Editor will look at/analyze/ 
change the cases that are 
WaitingForEdit. 

After the editor is done, the 
editor will assign a status of 
WasEdited or KeepAsIs to the 
case. KeepAsIs is assigned enly 
if there are rule violations which 
will not be changed. 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 






As the data were copied from the Collection Database to the Holding Database, a number of 
processes were run. Code-all-that-apply (COTA) recoding and “yes/no” recoding were applied. COTA 
recoding involved changing the multiple-response values of 0/1, 0/2, 0/3, etc., to a series of yes/no (1/2) 
responses. Yes/no recoding provided a means to resolve questions left unanswered in a series of yes/no 
items. If all marked answers were “Yes,” then the unanswered items were converted to “No.” However, if 
any item was “No,” “Don’t know,” or “Reftised,” all unanswered items were converted to -9 (Not 
ascertained). All blanks were converted to -9 (Not ascertained) and don’t know and reftised responses 
were converted to -7 and -8 as appropriate. It was at this stage that skip patterns were enforced using the 
Blaise CheckRules function and legitimate skips were assigned the standard code of-1 (see appendix Q: 
General Rules for Data Recoding Specifications During Hard-Copy Editing). 

Edit programs (range and logical checks) were run against all cases contained in the Holding 
Database. As the editing process continued, the Holding Database contained both new cases copied from 
the Collection Database and edited cases returned from the Editing Database (discussed later). Each case 
was assigned a status code that reflected its current status. For cases that were new to the Holding 
Database, the CheckRules function assigned one of two codes. The status CleanNew was assigned to new 
cases that contained no edit (range or logical) errors. The status DirtyNew was assigned to new cases that 
failed one or more edit checks. Those cases that had undergone edit updating were also subjected to edit 
checks to identify any errors that remained or were inadvertently introduced during edit updating. The 
CheckRules function assigned the status CleanEdit to cases with no remaining errors. The status DirtyEdit 
was assigned to cases returned from edit updating that had remaining or new errors. Those cases that were 
assigned a status of KeepAsIs in a previous editing round were considered clean. 

Cases that were found to have edit errors (DirtyNew and DirtyEdit) were copied to the 
Editing Database for review and updating. At that time, their status in the Holding Database was set to 
InEdit. A face sheet was generated for each case with editing errors, giving the batch number, case ID, 
and edit rules that had been violated. 



5.2.S.3 Editing Database 

Cases in the Holding Database that failed edit checks were copied to the Editing Database 
for the correction of errors. As cases were copied to the Editing Database, they were assigned a status of 
WaitingForEdit in the Editing Database. Editing staff worked from face sheets produced during the edit 
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checks conducted on the Holding Database to retrieve and correct case records. Using the batch number 
and case ID number, editors retrieved and reviewed hard-copy instruments as necessary to resolve editing 
errors. 



Once the editor had reviewed and updated each case as necessary, he or she assigned one of 
two outcome codes. The status code of WasEdited was assigned when all edit errors had been corrected. 
A status of KeepAsIs was assigned when the editor’s review indicated that data that violated an edit check 
should be retained, for example, when the hard-copy instrument indicated that an out-of-range value was 
correct. 



Cases with the statuses of WasEdited and KeepAsIs were moved back to the Holding 
Database. Cases that had a status of WasEdited in the Editing Database were assigned the status 
CheckEdit in the Holding Database. The edit rules were applied to these cases to ensure that they were 
clean. As noted earlier, cases assigned a status of KeepAsIs in the editing process were considered clean. 



5.2.S.4 Delivery Database 

The main purpose of the Delivery Database was to store the instrument data at the school, 
teacher, or child level in a “rectangular” format consistent with downstream activities in preparation for 
data delivery. Cases for which editing and coding activities were completed were copied from the 
Holding Database to the Delivery Database. These were cases with status codes of CleanNew, CleanEdit, 
or KeepAsIs. When the data were copied to the Delivery Database, the “Other, specify” opcodes and 
parent interview occupation codes were applied. See table 5-5 for a summary of the status codes assigned 
for data management databases. 



5.2.6 Data Editing 

The data editing process consisted of running range edits for soft and hard ranges, running 
consistency edits in Blaise, and then manually reviewing frequencies of the results. 
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Table 5-5. 



Status codes assigned for data management databases, third grade data collection: School 
year 2001-02 



Database Status codes assigned 

Collection CollectionNew: New cases 

CollectionCopied: Copied to Holding Database 

Holding New: New case copied from the Collection Database 

CheckEdit: Cases returned from the Editing Database to be subjected to edit checks again 
DirtyNew: New cases that fail edit check(s) 

DirtyEdit: Cases returned to from the Editing Database that have been edited and have 
failed edit check(s) 

CleanNew: New cases with no errors 
CleanEdit: Edited cases with no errors 
InEdit: Cases copied to the Editing Database 

KeepAsIs: Cases returned from the Editing Database that have been reviewed in the 
Editing Database and are not to be subjected to edits again 



Editing WaitingForEdit: Cases with errors waiting to be edited 

WasEdited: Cases for which editing is completed 

KeepAsIs: Edited cases for which the editor has determined that edit errors (e.g., out of 
range responses) should be retained as is 



Delivery InDelivery: Cases ready for delivery 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



5.2.6.1 Range Specifications 

Hard-copy range specifications set the parameters for high and low acceptable values for a 
question. Where values were printed on the forms, these were used as the range parameters. For open- 
ended questions, such as, “Counting this school year, how many years have you taught in your current 
school including part-time teaching?,” high and low ranges were established as acceptable values (see 
appendix R for range specifications for each questionnaire). Data frequencies were run on the range of 
values to identify any errors. Values outside the range were identified as errors and were printed for a data 
editor to review. Cases with range errors were identified, and the original response was updated. In some 
cases, range violations were retained in the data because the value was checked and found to be the value 
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reported by the teacher or school. These were marked as “KeepAsIs” cases. Data frequencies were then 
rerun and reviewed. This iterative process was repeated until no further range errors were found. 



S.2.6.2 Consistency Checks (Logical Edits) 

By programming logical edits between variables, consistency between variables not involved 
in a skip pattern was confirmed. For example, in the school administrator questionnaire, the number of 
children eligible for free breakfast could not exceed the total number of children enrolled in the school. 
These logical edits were run on the whole database after all data entry and range edits were complete (see 
appendix S: Data Edit Specifications for Each Questionnaire). The logical edits were run separately for 
each form. All batches of data were combined into one large data file, and data frequencies were 
produced. The frequencies were reviewed to ensure the data remained logically consistent within the 
form. When an inconsistency was found, the case was identified and the inconsistency was printed for an 
editor to review. The original value was corrected (or checked and “kept as is” if the data item was 
confirmed, and the case was again run through the consistency edits. Once the case passed the consistency 
edits, it was returned to the main data set. The frequencies were then rerun and reviewed. This was an 
iterative process; it was repeated until no further inconsistencies were found. Table 5-6 shows hard-copy 
questionnaire data preparation production. 



S.2.6.2. 1 Applying Data Retrieval in the Editing Process 

As described in section 4. 5. 5. 3, if critical items were missing from the hard-copy 
questionnaires, field supervisors were asked to contact the schools and collect this information using a 
data retrieval form. As completed critical item forms were received and receipted in the document control 
system, the data retrieval data were reviewed to see how it could be applied to the data collected in the 
original instruments. In general, the purpose of collecting data retrieval was to improve or complete the 
data collected in the original instrument. Flowever, that was not always the case. If, for example, data 
collected during data retrieval contradicted data already received, we would not use it. Appendix T: 
Decision Rules for Data Retrieval presents the decision rules that were used to apply the data retrieval 
data to each instrument. 
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Table 5-6. Hard-copy editing progress report, third grade data collection: School year 2001-02 



Instrument type 


Number of 
instruments 


Number of 
instruments clean 
after the first 
cleaning pass 


Number of 
instruments 
edited 


Instruments edited 

Number of Number of 

instruments clean instruments left 
from edit as is from edit 


Total 


38,681 


34,619 


4,062 


1,988 


2,074 


Percent 




89.5 


10.5 


5.1 


5.4 


Facilities checklist 
School administrator 


2,488 


2,487 


1 


1 


0 


questionnaire 


1,878 


730 


1,148 


469 


679 


School fact sheet 


1,890 


1,679 


211 


139 


72 


Special education A 


646 


633 


13 


9 


4 


Special education B 
Student records 


888 


753 


135 


75 


60 


abstract 

Teacher 


10,278 


9,918 


360 


260 


100 


questionnaire A 
Teacher 


4,401 


2,696 


1,705 


598 


1,107 


questionnaire B 
Teacher 


4,390 


4,220 


170 


127 


43 


questionnaire C 


11,822 


11,503 


319 


310 


9 



NOTE: The total in this table does not match the total receipted shown in table 5-4 because of refusals and other unusable 
questionnaires. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



S.2.6.2.2 Frequency and Cross-Tabulation Review 

As a final review, frequencies and cross-tabulations were run to determine consistency and 
accuracy within the various forms. If discrepancies could not be explained, no changes were made to the 
data. For example, in teacher questionnaire A, an item asking about languages other than English spoken 
in the classroom included a response option of “No language other than English.” If a respondent circled 
that response, but also answered (in subsequent items) that other languages besides English were spoken 
in the classroom, then the response was left as recorded by the respondent because the discrepancy could 
not be resolved. 
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6, RESPONSE RATES 



This chapter describes the computation of unit completion rates for the spring-third grade 
data collection of the ECLS-K, and unit overall response rates for the base year respondents. Weighted 
and unweighted unit completion rates are presented for different groups of children. Item response rates 
for selected items from the ECLS-K third grade instruments are also presented. 



6.1 Definition of Response and Completion Rates 

Response rates and completion rates are two ways to describe the outcomes of data 
collection activities. A response rate is the ratio of the number of units with completed interviews (for 
example, the units could be children, parents, schools or teachers) to the number of units sampled and 
eligible for the interview. The response rate indicates the percentage of possible interviews completed 
taking all survey stages into account. On the other hand, the completion rate measures the percentage of 
interviews completed for a specific stage of the survey. For example, in the base year of the ECLS-K 
children were identified for assessment in a two-stage process. The first stage involved the recruitment of 
schools to participate in the study. Preassessment visits were made to schools that agreed to participate. 
During the preassessment visit, field supervisors met with the participating school’s school coordinator to 
enumerate and sample the kindergartners. Assessments were then conducted for the sampled children whose 
parents consented. If the school refused to participate in the study, no children were sampled for 
assessments. Under this design, the completion rate for the child assessment is the percentage of sampled 
children who completed the assessment. The response rate is the product of the school participation or 
cooperation rate and the child assessment completion rate. 

Response and completion rates can be either unweighted or weighted. The unweighted rate, 
computed using the raw number of cases, provides a useful description of the success of the operational 
aspects of the survey. The weighted rate, computed by summing the weights (usually the reciprocals of 
the probability of selecting the units) for both the numerator and denominator, gives a better description 
of the success of the survey with respect to the population sampled since the weights allow for inference 
of the sample data (including response status) to the population. Both rates are usually not very different 
unless the probabilities of selection and the response rates in the categories with different selection 
probabilities vary considerably. 
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For example, the completion rate of the ECLS-K child assessment (CA) is computed as 



rcA 



z 

Z Wi 



i ^ERfjA '-jENRca 



where Wi is the weight (inverse of the probability of selection of the child) for child i, and ERca denotes 
eligible child assessment respondent and ENRqa eligible child assessment nonrespondent. To compute the 
unweighted rates, Wi is set to 1 for each child. 

The response rate of the child assessment can be computed as 



^CA - fs” ^ At 



where rs is the school cooperation rate and rcA is the child assessment completion rate. 

After the base year, only completion rates were computed for the different ECLS-K 
instruments, since the response rates of the schools where the children were sampled remained the same 
for each subsequent round. Data users can compute the third grade response rate for each ECLS-K 
instrument by multiplying the school response rate from the base year and the third grade completion rate 
for each instrument. 



6.2 Completion Rates 

For the ECLS-K third grade data collection, there were 11 survey instruments: child 
assessment; parent interview; teacher questionnaire part A, part B, and part C; special education teacher 
questionnaire part A and part B; school administrator questionnaire; school fact sheet; facilities checklist; 
and student records abstract. Except for the child assessment and parent interview, all other instruments 
were paper-and-pencil instruments. For each instrument, completion rates were computed separately for 
children who were sampled as part of the kindergarten cohort in the base year and for children who were 
sampled in first grade through the student sample freshening procedure. These rates were then combined 
to obtain the completion rates for all children in first grade. 
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6.2,1 



Children Sampled in Kindergarten 



For the ECLS-K, a completion rate is a response rate conditioned on the results of an earlier 
stage of data collection. In the case of the ECLS-K third grade data collection, the condition is that 
children who were sampled in kindergarten were base year respondents since only base year respondents 
were eligible for subsequent data collection efforts. Children sampled in first grade were exempt from this 
condition in the computation of completion rates. They are discussed in section 6.2.2. 

For each instrument, the unweighted completion rate is the proportion of base year 
respondents with completed data for the instrument to the base year respondents who remain eligible to 
have the third grade instrument administered. Base year respondents who were subsampled out because 
they moved from their base year original sample schools and base year respondents who died or moved 
out of the country were not included in the denominator. For the weighted completion rates, the weight 
used is the product of the school base weight, the within-school child weight, and the factor that was used 
to adjust for movers between base year and third grade who were subsampled out for data collection. For 
a description of these weights and adjustment factor, see chapter 7. 

Tables 6-1 to 6-4 present weighted and unweighted child-level completion rates for spring- 
third grade data collection, broken out by school characteristics. These rates pertain to children who were 
sampled as part of the kindergarten cohort in the base year. (Rates for students sampled in first grade 
through the student sample freshening procedure can be found in table 6-14.) 

Relative to spring-first grade the overall completion rates for the child assessments (80.8 
percent) and the parent interview (77.8 percent), shown in table 6.1, both decreased 7 percentage points in 
spring-third grade. See chapter 5 of the Early Childhood Longitudinal Study, Kindergarten Class of 1998- 
99 {ECLS-K) User’s Manual for the ECLS-K First Grade Public-Use Data Files and Electronic Code 
Book (NCES 2002-135; U.S. Department of Education, National Center for Education Statistics, 2002) 
for a complete set of spring-first grade completion rates. The decrease is almost completely due to the 
increase in the number of children who moved outside of the sampled primary sampling units (PSUs) or 
moved within the sampled PSUs but could not be located (the numbers slightly more than doubled in both 
cases).' These children are included in the category labeled “Unknown” for each of the different school 

* Assessment was not attempted for children who moved outside of the sample PSUs, but the parents of these children were contacted by 
telephone and asked to respond to the parent interview. These children were included in the computation of the completion rates for all 
instruments. Similarly, children who were subsampled to be followed into their new schools but could not be located were obviously not 
assessed, but parent interviews were attempted as in the case of children who moved outside of sampled PSUs. 
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characteristics (tables 6-1 to 6-4). The category “unknown” also includes 49 children who were 
homeschooled and thus had no information concerning schools. As shown in table 6.1, the completion 
rates for the child assessments are quite high and uniform across school characteristics (ranging from 97.1 
percent in schools with 750 or more enrolled to 100 percent in schools in large towns), except for the 
“unknown” category for the different school characteristics. Similarly, the completion rates for the parent 
interviews were fairly consistent across school characteristics (ranging from 78.5 percent for children in 
schools with more than 90 percent non- White enrolled to 93.4 percent for children in large towns, with an 
overall completion rate of 78.5 percent), except for the “unknown” category. The “unknown” category 
aside, both the child assessments and the parent interview completion rates increased between spring-first 
grade and spring-third grade for all school characteristics. The completion rates by mover status are 
discussed later, but the rates of completing all the instruments are much lower for children who moved 
than for those who did not move. 

In all tables, the unweighted completion rates are almost always higher than the weighted 
completion rates, by as much as 7 percent at the overall level. This difference is due to movers who have 
larger weights and higher nonresponse rates than nonmovers. The weights of the movers were increased 
to account for the subsampling of movers. They also responded at a much lower rate than nonmovers, as 
shown in tables 6-5 to 6-8 and discussed later. 

Table 6-2 shows that the overall weighted completion rate is 66.1 percent for the school 
administrator questionnaire and 76.5 percent for the school fact sheet. The completion rate for the school 
fact sheet is about 1 0 percent higher than the school administrator questionnaire due to the extension of 
data collection into fall 2002 that affected the school fact sheet and the student records abstract. The 
completion rate for the school administrator questionnaire, 66.1 percent, is about 10 percentage points 
lower than the corresponding rate in spring-first grade, 76.3 percent (see chapter 5 of the User’s Manual 
for the ECLS-K First Grade Public-Use Data Files and Flectronic Code Book (NCES 2002-135) for 
spring-first grade completion rates). Note that there was no school fact sheet in spring-first grade. Once 
again, the increase in the movers is largely responsible for the lower rates, as discussed below. In third 
grade, 9,889 children no longer attended the schools where they were sampled in kindergarten. Of these, 
41 percent were not subsampled. By comparison, 5,612 children in the spring-first grade sample no longer 
attended the original sample schools, and 49 percent of them were not subsampled. The subsampling rate 
is higher in third grade than in first grade because language minority children were followed at a much 
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higher rate, as discussed in chapter 3. The mover rate was 26 percent in spring-first grade and 46 percent 
in spring-third grade.^ 

Table 6-2 shows that the completion rates for the school administrator questionnaire range 
from 69.0 percent for children in large cities to 99.4 percent for those in large towns (ignoring the 
“unknown” category). Rates for the school fact sheet follow the same pattern. Excluding the movers, the 
completion rate for the school administrator questionnaire is much higher, with an overall rate of 86.7 
percent as shown in table 6-6, only slightly lower than the spring-first grade rate of 88.7 percent. In the 
case of the school fact sheet, the rate for nonmovers is 95.8 percent. It is worth noting that the completion 
rates for the school administrator questionnaire are lower for schools with higher percentages of 
minorities, a phenomenon also observed in previous rounds for the school administrator questionnaire. 
However, this disparity decreased considerably in the first-grade year and in spring-third grade compared 
to the base year, reflecting the success of increased data collection efforts targeted toward these schools. 
See chapter 4 for a discussion of the followup for missing school administrative questionnaires. The 
facilities checklist has a 77.5 percent completion rate, which is about 1 1 points higher than that for the 
school administrator questionnaire, but only 1 point higher than the rate for the school fact sheet. The 
student records abstract, which was to have been completed for all students except for those who moved 
and could not be located, had a 67.0 percent completion rate. Excluding children whose school 
characteristics are unknown, the completion rates for the facilities checklist in table 6-3 range from 89.1 
percent for children in the suburbs of mid-size cities to 100 percent for children in large towns. For the 
student records abstract, the completion rates are lower, ranging from 72 percent for children in large 
cities to 95.8 percent for children in large towns. Table 6-7 shows the completion rates for these two 
instruments, separately for movers and nonmovers. Excluding the movers, the completion rates for both 
instruments are much higher. The rate for the facilities checklists is 97.2 percent for nonmovers compared 
with 50 percent for movers. The rate for the student records abstracts is 85.2 percent for nonmovers 
compared with 41.6 percent for movers. 

All three of the teacher questionnaires were completed at an overall rate of 62 to 63 percent. 
The completion rates have substantial variation when broken out by school characteristics, even when the 
“unknown” category for the school characteristics is ignored. The completion rates are 90 percent or more 
for Catholic schools and for schools in large and small towns. Schools in large cities, schools with 750 
students or more, and schools with 90 percent or more minority enrollment have completion rates in the 

^ The number of movers and the mover rates are slightly different than those discussed in chapter 3. For the discussion of sampling issues in 
chapter 3, children who attended schools terminating in a high grade who then moved into destination schools were not considered movers. 
Operationally, these children were all followed into their new schools. 
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60s. The “unknown” categories have, by far, the lowest completion rates. For a discussion of how data for 
the different instruments were collected for children who moved from their previous round schools, see 
chapter 4. 
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Table 6-1. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year 







Child assessments 






Parent interview 




School characteristics^ 




Completion rates 




Completion rates 






Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted Unweighted 


All school types 


14,349 


80.8 


86.1 


13,392 


77.8 


80.3 


School type 


Public 


11,671 


98.1 


98.5 


10,268 


86.3 


86.6 


Private 


2,629 


98.7 


99.2 


2,452 


92.6 


92.5 


Catholic 


1,662 


99.0 


99.3 


1,546 


92.9 


92.4 


Other private 


967 


98.1 


99.0 


906 


92.3 


92.7 


Unknown school type 


49 


2.6 


2.3 


672 


36.2 


31.0 


Type of locale 


Large city 


2,431 


97.8 


98.3 


2,038 


80.9 


82.4 


Mid-size city 


2,474 


98.9 


99.3 


2,216 


88.6 


88.9 


Urban fringe of large city 


4,169 


97.4 


98.0 


3,723 


86.9 


87.5 


Urban fringe of mid-size city 


942 


97.7 


98.0 


832 


86.2 


86.6 


Large town 


375 


100.0 


100.0 


355 


93.4 


94.7 


Small town 


1,032 


99.4 


99.2 


967 


90.0 


93.0 


Rural - Outside MSA 


1,498 


98.8 


99.4 


1,350 


89.7 


89.6 


Rural - Inside MSA 


1,201 


98.2 


98.7 


1,075 


88.0 


88.3 


Unknown 


227 


8.5 


9.7 


836 


39.5 


35.6 


School size (total enrollment) 


1 to 299 


3,078 


98.2 


98.8 


2,809 


89.8 


90.2 


300 to 499 


4,562 


98.5 


98.9 


4,118 


88.5 


89.2 


500 to 749 


4,043 


98.5 


98.7 


3,569 


87.3 


87.2 


750 or more 


2,505 


97.1 


97.9 


2,116 


81.4 


82.7 


Unknown 


161 


6.2 


7.0 


780 


38.3 


34.1 



See notes at end of table. 



Table 6-1. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year — Continued 







Child assessment 






Parent interview 




School characteristics^ 




Completion rates 




Completion rates 






Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted Unweighted 


Percent non- White enrolled 


0-10 


4,580 


99.3 


99.4 


4,269 


92.3 


92.7 


11-49 


4,594 


97.5 


98.2 


4,179 


88.3 


89.4 


50-89 


2,564 


98.0 


98.4 


2,197 


83.8 


84.3 


90-100 


2,412 


97.8 


98.2 


1,932 


78.5 


78.6 


Unknown 


199 


7.3 


8.6 


815 


38.9 


35.1 


Region 


Northeast 


2,667 


98.6 


98.8 


2,411 


88.8 


89.3 


Midwest 


3,677 


98.1 


98.5 


3,405 


90.8 


91.2 


South 


4,674 


97.6 


98.3 


4,038 


83.5 


85.0 


West 


3,291 


98.6 


98.8 


2,880 


87.2 


86.5 


Unknown 


40 


2.2 


1.9 


658 


35.8 


30.5 



' School characteristics are for schools attended by children in the ECLS-K third grade sample and are based on ECLS-K survey data, not data from the sampling frame. 

^ Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 



Table 6-2. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year 





School administrator questionnaire 






School fact sheet 




School characteristics^ 




Completion rates 






Completion rates 




Completes^ 


Weighted Unweighted 


Completes^ 


Weighted 


Unweighted 


All school types 


12,361 


66.1 


73.3 


14,064 


76.5 


83.4 


School type 


Public 


9,559 


78.4 


80.7 


10,976 


91.1 


92.6 


Private 


2,352 


85.8 


88.7 


2,545 


94.6 


96.0 


Catholic 


1,515 


88.5 


90.5 


1,623 


95.9 


97.0 


Other private 


837 


81.4 


85.7 


922 


92.5 


94.4 


Unknown school type 


450 


12.0 


19.0 


543 


14.5 


22.9 


Type of locale 


Large city 


1,832 


69.0 


74.1 


2,246 


88.6 


90.8 


Mid-size city 


2,138 


82.2 


85.8 


2,349 


93.1 


94.3 


Urban fringe of large city 


3,271 


73.9 


76.9 


3,894 


89.7 


91.6 


Urban fringe of mid-size city 


792 


78.0 


82.4 


903 


91.0 


94.0 


Large town 


374 


99.4 


99.7 


375 


100.0 


100.0 


Small town 


933 


88.0 


89.7 


1,015 


97.2 


97.6 


Rural - outside MSA 


1,401 


93.0 


93.0 


1,480 


97.7 


98.2 


Rural - inside MSA 


1,053 


81.7 


86.5 


1,132 


89.9 


93.0 


Unknown 


567 


15.3 


22.2 


670 


17.4 


26.3 


School size (total enrollment) 


1 to 299 


2,883 


90.0 


92.6 


3,024 


95.9 


97.1 


300 to 499 


3,934 


81.9 


85.2 


4,328 


92.4 


93.8 


500 to 749 


3,234 


76.5 


79.0 


3,807 


90.4 


93.0 


750 or more 


1,859 


72.8 


72.6 


2,320 


90.2 


90.6 


Unknown 


451 


11.6 


18.1 


585 


15.5 


23.5 



See notes at end of table. 
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Table 6-2. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year — Continued 



School administrator questionnaire School fact sheet 



School characteristics^ 




Completion rates 




Completion rates 




Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


Percent non- White enrolled 


0-10 


4,017 


85.2 


87.2 


4,412 


94.6 


95.8 


11-49 


3,973 


82.1 


85.0 


4,379 


91.1 


93.6 


50-89 


1,970 


73.1 


75.6 


2,389 


90.6 


91.6 


90-100 


1,921 


74.0 


78.2 


2,265 


90.9 


92.2 


Unknown 


480 


12.2 


19.0 


619 


16.2 


24.5 


Region 


Northeast 


2,035 


72.8 


75.4 


2,460 


89.5 


91.1 


Midwest 


3,349 


86.9 


89.7 


3,575 


94.4 


95.7 


South 


3,864 


79.0 


81.3 


4,454 


91.4 


93.7 


West 


2,668 


76.0 


80.1 


3,039 


89.9 


91.3 


Unknown 


445 


11.8 


18.9 


536 


14.4 


22.8 



’ School characteristics are for schools attended by children in the ECLS-K third grade sample and are based on ECLS-K survey data, not data from the sampling frame. 

^ A completed questionnaire was defined as one that was not completely left blank. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 
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Table 6-3. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year 





Lacilities checklist 




Student records abstract 




School characteristics^ 




Completion rates 




Completion rates 






Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted Unweighted 


All school types 


14,280 


77.5 


84.7 


12,359 


67.0 


73.3 


School type 


Public 


11,148 


92.5 


94.1 


9,907 


82.2 


83.6 


Private 


2,602 


96.4 


98.2 


2,402 


89.2 


90.6 


Catholic 


1,646 


96.7 


98.3 


1,560 


92.1 


93.2 


Other private 


956 


96.0 


97.9 


842 


84.4 


86.2 


Unknown school type 


530 


13.7 


22.4 


50 


1.3 


2.1 


Type of locale 


Large city 


2,361 


93.5 


95.4 


1,866 


72.0 


75.4 


Mid-size city 


2,392 


94.4 


96.0 


2,083 


81.3 


83.6 


Urban fringe of large city 


3,930 


90.0 


92.4 


3,544 


80.6 


83.3 


Urban fringe of mid-size city 


886 


89.1 


92.2 


861 


86.3 


89.6 


Large town 


375 


100.0 


100.0 


362 


95.8 


96.5 


Small town 


1,029 


98.2 


98.9 


971 


91.1 


93.4 


Rural - Outside MSA 


1,464 


96.8 


97.1 


1,418 


93.7 


94.1 


Rural - Inside MSA 


1,142 


91.4 


93.8 


1,093 


88.6 


89.8 


Unknown 


701 


18.3 


27.5 


161 


5.2 


6.3 


School size (total enrollment) 


1 to 299 


3,031 


96.1 


97.3 


2,878 


90.6 


92.4 


300 to 499 


4,408 


93.7 


95.5 


4,092 


86.3 


88.7 


500 to 749 


3,860 


92.9 


94.3 


3,405 


81.3 


83.2 


750 or more 


2,364 


90.3 


92.3 


1,899 


75.5 


74.2 


Unknown 


617 


15.7 


24.8 


85 


2.4 


3.4 



See note at end of table. 
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Table 6-3. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year — Continued 



School characteristics^ 


Facilities checklist 




Student records abstract 




Completion rates 




Completion rates 


Completes^ 


Weighted Unweighted 


Completes^ 


Weighted 


Unweighted 


Percent non- White enrolled 














0-10 


4,438 


94.8 


96.4 


4,318 


91.4 


93.7 


11-49 


4,385 


91.6 


93.8 


3,986 


83.9 


85.2 


50-89 


2,444 


92.5 


93.7 


2,100 


79.5 


80.6 


90-100 


2,375 


95.1 


96.7 


1,837 


72.9 


74.8 


Unknown 


638 


16.0 


25.3 


118 


3.2 


4.7 


Region 














Northeast 


2,528 


91.1 


93.7 


2,187 


79.3 


81.0 


Midwest 


3,644 


96.6 


97.6 


3,414 


88.8 


91.4 


South 


4,442 


91.7 


93.5 


4,058 


83.9 


85.4 


West 


3,144 


92.4 


94.4 


2,658 


77.6 


79.8 


Unknown 


522 


13.5 


22.2 


42 


1.0 


1.8 



' School characteristics are for schools attended by children in the ECLS-K third grade sample and are based on ECLS-K survey data, not data from the sampling frame. 

^ Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 
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Table 6-4. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year 





Teacher questionnaire part A 




Teacher questionnaire part B 


Teacher questionnaire part C 


School characteristics’ 




Completion rates 




Completion rates 




Completion rates 




Completes^ 


Weighted Unweighted 


Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


All school types 


11,770 


62.4 


69.8 


11,741 


62.3 


69.6 


11,802 


62.7 


70.0 


School type 


Public 


9,274 


75.7 


78.3 


9,242 


75.5 


78.0 


9,301 


76.0 


78.5 


Private 


2,416 


87.3 


91.1 


2,418 


87.2 


91.2 


2,425 


87.4 


91.5 


Catholic 


1,557 


90.2 


93.0 


1,564 


90.5 


93.4 


1,575 


91.4 


94.1 


Other private 


859 


82.7 


87.9 


854 


82.0 


87.4 


850 


81.3 


87.0 


Unknown school type 


80 


2.2 


3.4 


81 


2.3 


3.4 


76 


2.2 


3.2 


Type of locale 


Large city 


1,649 


62.0 


66.7 


1,647 


62.0 


66.6 


1,654 


62.4 


66.9 


Mid-size city 


2,087 


80.0 


83.7 


2,079 


79.5 


83.4 


2,096 


80.2 


84.1 


Urban fringe of large city 


3,322 


74.5 


78.1 


3,295 


73.9 


77.5 


3,329 


74.6 


78.3 


Urban fringe of mid-size 


801 


78.1 


83.4 


804 


78.9 


83.7 


808 


79.5 


84.1 


city 


Large town 


363 


96.0 


96.8 


370 


98.2 


98.7 


368 


97.7 


98.1 


Small town 


985 


92.9 


94.7 


982 


92.8 


94.4 


977 


92.1 


93.9 


Rural - outside MSA 


1,356 


85.3 


90.0 


1,363 


85.8 


90.4 


1,369 


86.6 


90.8 


Rural - inside MSA 


1,027 


80.1 


84.4 


1,020 


79.4 


83.8 


1,023 


79.5 


84.1 


Unknown 


180 


5.4 


7.1 


181 


5.5 


7.1 


178 


5.4 


7.0 


School size (total enrollment) 


1 to 299 


2,832 


87.9 


90.9 


2,817 


87.3 


90.5 


2,829 


87.8 


90.8 


300 to 499 


3,911 


81.2 


84.7 


3,920 


81.4 


84.9 


3,924 


81.5 


85.0 


500 to 749 


3,181 


75.3 


77.7 


3,164 


75.0 


77.2 


3,199 


75.8 


78.1 


750 or more 


1,730 


65.1 


67.6 


1,723 


65.0 


67.3 


1,735 


65.6 


67.8 


Unknown 


116 


3.2 


4.7 


117 


3.3 


4.7 


115 


3.3 


4.6 



See notes at end of table. 
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Table 6-4. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by school 
characteristic — children sampled in the base year — Continued 



Teacher questionnaire part A Teacher questionnaire part B Teacher questionnaire part C 

School characteristics' Completion rates Completion rates Completion rates 





Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


Percent non- White enrolled 


0-10 


4,216 


88.2 


91.5 


4,206 


87.9 


91.3 


4,211 


88.0 


91.4 


11-49 


3,839 


77.9 


82.1 


3,835 


77.9 


82.0 


3,872 


78.6 


82.8 


50-89 


1,963 


73.4 


75.3 


1,951 


73.2 


74.8 


1,960 


73.9 


75.2 


90 - 100 


1,605 


62.2 


65.3 


1,601 


62.0 


65.2 


1,611 


62.3 


65.6 


Unknown 


147 


4.0 


5.8 


148 


4.0 


5.9 


148 


4.1 


5.9 


Region 


Northeast 


2,168 


77.1 


80.3 


2,155 


76.7 


79.8 


2,170 


77.8 


80.4 


Midwest 


3,272 


85.0 


87.6 


3,275 


85.1 


87.7 


3,292 


85.3 


88.2 


South 


3,822 


76.2 


80.4 


3,815 


75.9 


80.3 


3,842 


76.6 


80.8 


West 


2,433 


69.2 


73.1 


2,420 


68.9 


72.7 


2,427 


69.1 


72.9 


Unknown 


75 


2.0 


3.2 


76 


2.1 


3.2 


71 


2.0 


3.0 



’ School characteristics are for schools attended by children in the ECLS-K third grade sample and are based on ECLS-K survey data, not data from the sampling frame. 

^ A completed questionnaire was defined as one that was not completely left blank. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 



As noted earlier, the rate at which the survey instruments were completed varies markedly 
by mover status and within movers, by whether or not the child was located and followed. As shown in 
table 6-5, the completion rate for the child assessments was 94.5 percent for children still enrolled in their 
base year school. For movers it dropped by close to 9 points to 85.6 percent for those who were located 
and followed, and for those not located or followed due to a move to a non-ECLS-K PSU, it was zero. 
The parent interview completion rates varied from 84.5 percent for nonmovers to 74.8 percent for movers 
who were located and followed for the purposes of the child assessments, to 5 1 .2 percent for movers who 
could either not be located or were not followed for the purposes of the child assessments. Even though 
children who had moved to a non-ECES-K PSU were not administered the child assessments, a parent 
interview was conducted by telephone wherever possible leading to the 51.2 percent response rate for this 
category. 



Table 6-6 shows that the school administrator questionnaire completion rate is about 30 
points lower for movers compared to nonmovers, even when the children who had moved were located 
and followed. For the school fact sheet, it is about 20 points lower for movers than for nonmovers. The 
facilities checklist (see table 6-7) behaves very much like the school fact sheet while the student records 
abstract has rates that are closer to those for the school administrative questionnaire. There are several 
reasons for the differences in rates between movers and nonmovers: located movers were not always 
assessed in schools; new schools in which movers enrolled had a lower level of commitment to the 
ECES-K and often refused to complete the school administrator questionnaire;^ and, some of these 
schools were contacted too late in the school year for them to consider completing it. For example, a child 
was identified as moving into another school during the last weeks of the field period; since the school 
year was over, the school and teacher questionnaires could not be collected for the child in that new 
school. The completion rate for nonmovers was 86.7 percent for the school administrator questionnaire 
and 95.8 percent for the school fact sheet. For located and followed movers it was 56.0 and 74.5 percent 
for the school administrator questionnaire and the school fact sheet, respectively. 

For all three teacher questionnaires the completion rates were approximately 82 percent if 
the child had not moved; about 54 percent if the child moved, was located, and followed; and just about 0 
if not located or followed (table 6-8). There was a handful of children who could not be located but for 
whom teacher data were available. These data were collected from teachers in the schools from which the 
children transferred, facilitated by the teacher-child link identified in the fall of the school year. The 



^ Of the 3,228 schools in the third grade sample, 41 percent refused to complete this instrument. 
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reasons for lower completion rates from teachers if the child moved are similar to the reasons that 
affected the school administrator questionnaire and school fact sheet completion rates for movers. 
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Table 6-5. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, 
by child’s mover status — children sampled in the base year 



Mover status’ 


Child assessments 




Parent interview 


Completes^ 


Completion rates 


Completes^ 


Completion rates 


Weighted 


Unweighted 


Weighted 


Unweighted 


All students 


14,349 


80.8 


86.1 


13,392 


77.8 


80.3 


Mover status 














Mover 


2,791 


60.9 


62.6 


3,063 


68.0 


68.8 


Located, followed 


2,791 


85.6 


85.1 


2,451 


74.8 


74.7 


Other"’ 


0 


0.0 


0.0 


612 


51.2 


52.1 


Nonmover 


11,558 


94.5 


94.6 


10,329 


84.5 


84.6 



' This is the mover status used in weighting, which does not consider children who moved into identified destination schools as movers. A destination school is 
a school that received at least four students from the school where they had just completed the highest grade. 

^ Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

"^This category includes movers who could not be located, and movers who moved into nonsampled PSUs. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade 
data collection, school year 2001-02. 
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Table 6-6. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, 
by child’s mover status — children sampled in the base year 



Mover status’ 


School administrator questionnaire 




School fact sheet 


Completes^ 


Completion rates 


Completes^ 


Completion rates 


Weighted 


Unweighted 


Weighted 


Unweighted 


All students 


12,361 


66.1 


73.3 


14,064 


76.5 


83.4 


Mover status 














Mover 


1,739 


37.2 


37.4 


2,327 


49.6 


50.0 


Located, followed 


1,739 


56.0 


54.5 


2,327 


74.5 


72.9 


Othef 


0 


0.0 


0.0 


0 


0.0 


0.0 


Nonmover 


10,622 


86.7 


87.0 


11,737 


95.8 


96.1 



' This is the mover status used in weighting, which does not consider children who moved into identified destination schools as movers. A destination school is 
a school that received at least four students from the school where they had just completed the highest grade. 

^ A completed questionnaire was defined as one that was not completely left blank. 

^ This category includes movers who could not be located, and movers who moved into nonsampled PSUs. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third 
grade data collection, school year 2001-02. 
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Table 6-7. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, 
by child’s mover status — children sampled in the base year 



Mover status’ 


Facilities checklist 


Student records abstract 


Completes^ 


Completion rates 


Completes^ 


Completion rates 


Weighted 


Unweighted 


Weighted 


Unweighted 


All students 


14,280 


77.5 


84.7 


12,359 


67.0 


73.3 


Mover status 














Mover 


2,368 


50.0 


50.9 


1,952 


41.6 


41.9 


Located, followed 


2,368 


75.2 


74.2 


1,952 


62.6 


61.1 


Other^ 


0 


0.0 


0.0 


0 


0.0 


0.0 


Nonmover 


11,912 


97.2 


97.5 


10,407 


85.2 


85.2 



' This is the mover status used in weighting, which does not consider children who moved into identified destination schools as movers. A destination school 
is a school that received at least four students from the school where they had just completed the highest grade. 

^ A completed questionnaire was defined as one that was not completely left blank. 

^ This category includes movers who could not be located, and movers who moved into nonsampled PSUs. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third 
grade data collection, school year 2001-02. 
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Table 6-8. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by child’s mover 
status — children sampled in the base year 



Teacher questionnaire part A Teacher questionnaire part B Teacher questionnaire part C 

Mover status Completion rates Completion rates Completion rates 





Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


All students 
Mover status 


11,770 


62.4 


69.8 


11,741 


62.3 


69.6 


11,802 


62.7 


70.0 


Mover 


1,656 


35.7 


35.6 


1,645 


35.6 


35.3 


1,650 


35.7 


35.5 


Located, followed 


1,653 


53.7 


51.8 


1,641 


53.4 


51.4 


1,646 


53.7 


51.6 


Other^ 


3 


0.1 


0.2 


4 


0.2 


0.3 


4 


0.2 


0.3 


Nonmover 


10,114 


81.5 


82.8 


10,096 


81.4 


82.7 


10,152 


81.9 


83.1 



' This is the mover status used in weighting, which does not consider children who moved into identified destination schools as movers. A destination school is a school that received at least four 
students from the school where they had just completed the highest grade. 

^ A completed questionnaire was defined as one that was not completely left blank. 

^ This category includes movers who could not be located, and movers into nonsampled PSUs. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, school year 
2001 - 02 . 



Tables 6-9 to 6-12 present child-level completion rates for the spring- third grade data 
collection, this time broken out by child characteristics for children who were sampled as part of the 
kindergarten cohort in the base year. Child characteristics are from the survey data. Gender, race/ethnicity 
and year of birth are all composite variables found in the data fde (see chapter 7 of the Early Childhood 
Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) User’s Manual for the Third Grade 
Public-Use Data File and Electronic Code Book (NCES 2004-001; U.S. Department of Education, 
National Center for Education Statistics, 2004) for a description of composite variables). For respondents, 
data are from third grade; for nonrespondents, data are from previous rounds. For the child assessments 
the completion rate was highest for Asians and Pacific Islanders (84.1 percent and 84.9 percent 
respectively) and lowest for American Indians or Alaska Natives (75.5 percent). For the parent interview 
it was highest for Whites (82.9 percent) and lowest for Blacks (67.0 percent) and Asian students (68.6 
percent). The ECES-K sample of Pacific Islanders is very clustered and has unusually high completion 
rates for the instruments filled out by school personnel, 80.9 percent for the school administrator 
questionnaire, 84.6 percent for the school fact sheet, and more than 73 percent for each of the teacher 
questionnaires. The lowest completion rate for the school administrator questionnaire is for Blacks (57.1 
percent); for the school fact sheet, it is for American Indians or Alaska Natives (66.0 percent). For the 
teacher questionnaires the lowest rates are in the 53 to 55 percent range and are associated with Blacks, 
Hispanics, and American Indians or Alaska Natives. Since 60 percent of the Black and Hispanic students 
fielded in spring-third grade are enrolled in high minority schools (50 percent or higher), this may be 
associated with lower levels of response for the school administrator questionnaire from high minority 
schools. Of the 32 percent of Black and Hispanic students with no school administrator questionnaire 
data, roughly half are enrolled in high minority schools. 
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Table 6-9. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by child 
characteristic — children sampled in the base year 







Child assessment 






Parent interview 




Child characteristics’ 




Completion rates 




Completion rates 






Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted Unweighted 


All students 


14,349 


80.8 


86.1 


13,392 


77.8 


80.3 


Sex"* 


Male 


7,285 


80.3 


85.7 


6,858 


78.3 


80.7 


Female 


7,064 


81.5 


86.6 


6,534 


77.3 


80.1 


Race/ethnicity 


White (not Hispanic) 


8,119 


81.2 


86.9 


8,000 


82.9 


85.6 


Black (not Hispanic) 


1,872 


78.0 


83.6 


1,570 


67.0 


70.2 


Hispanic 


2,575 


82.2 


85.9 


2,295 


73.9 


76.6 


Asian 


963 


84.1 


86.4 


757 


68.6 


67.9 


Pacific Islander 


171 


84.9 


87.2 


161 


82.3 


82.1 


American Indian or Alaska Native 


248 


75.5 


81.3 


242 


75.5 


79.3 


Other 


382 


81.7 


88.4 


360 


78.8 


83.3 


Unknown race/ethnicity 


19 


42.9 


48.7 


7 


7.8 


17.9 


Year of birth 


1992 


4,179 


80.5 


85.7 


3,922 


77.3 


80.5 


1993 


10,086 


81.1 


86.4 


9,397 


78.2 


80.5 


Other/unknown 


84 


59.6 


71.2 


73 


53.3 


61.9 



' Based on ECLS-K survey data and not on data from the sampling frame. 

^ Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

There is a small number of children whose gender is unknown and who did not have completed child assessment and parent interview. The completion rates for these children, being 
zero, are not included in the table. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 
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Table 6-10. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by child 
characteristic — children sampled in the base year 





School administrator questionnaire 






School fact sheet 




Child characteristics’ 




Completion rates 






Completion rates 






Completes^ 


Weighted Unweighted 


Completes^ 


Weighted Unweighted 


All students 


12,361 


66.1 


73.3 


14,064 


76.5 


83.4 


Sex^ 


Male 


6,241 


65.0 


72.6 


7,119 


75.7 


82.9 


Female 


6,108 


67.2 


73.9 


6,933 


77.3 


83.9 


Unknown sex 


12 


100.0 


100.0 


12 


100.0 


100.0 


Race/ethnicity 


White (not Hispanic) 


7,309 


70.8 


78.2 


8,061 


79.1 


86.2 


Black (not Hispanic) 


1,489 


57.1 


64.6 


1,810 


70.5 


78.5 


Hispanic 


2,062 


61.1 


66.5 


2,484 


75.2 


80.1 


Asian 


786 


64.9 


69.6 


920 


78.1 


81.4 


Pacific Islander 


172 


80.9 


86.9 


178 


84.6 


89.9 


American Indian or Alaska 


212 


61.9 


67.1 


221 


66.0 


69.9 


Native 


Other 


305 


60.4 


70.3 


358 


72.4 


82.5 


Unknown race/ethnicity 


26 


68.2 


66.7 


32 


83.7 


82.1 


Year of birth 


1992 


3,681 


68.1 


74.8 


4,115 


76.9 


83.6 


1993 


8,602 


65.4 


72.7 


9,856 


76.4 


83.3 


Other/unknown 


78 


54.5 


65.0 


93 


69.4 


77.5 



^ Based on ECLS-K survey data and not on data from the sampling frame. 

^ A completed questionnaire was defined as one that was not completely left blank. 

^The completion of the school-level instruments does not depend on whether the child has completed assessment or parent interview data. Hence, while all children with completed 
assessment or parent interview data have known value of gender; there are children with completed school-level data whose gender is unknown. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 
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Table 6-11. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by child 
characteristic — children sampled in the base year 







Facilities checklist 






Student records abstract 


Child characteristics’ 




Completion rates 




Completion rates 




Completes^ 


Weighted 


Unweighted 


Completes^ 


Weighted 


Unweighted 


All students 


14,280 


77.5 


84.7 


12,359 


67.0 


73.3 


Sex^ 


Male 


7,224 


76.8 


84.1 


6,260 


66.5 


72.9 


Female 


7,055 


78.5 


85.3 


6,099 


67.8 


73.8 


Unknown sex 


1 


1.6 


8.3 


0 


0.0 


0.0 


Race/ethnicity 


White (not Hispanic) 


8,084 


79.3 


86.5 


7,337 


71.5 


78.5 


Black (not Hispanic) 


1,877 


73.4 


81.4 


1,518 


59.5 


65.9 


Hispanic 


2,545 


76.8 


82.1 


2,044 


62.4 


65.9 


Asian 


952 


79.4 


84.2 


766 


64.7 


67.8 


Pacific Islander 


180 


85.8 


90.9 


149 


70.8 


75.3 


American Indian or Alaska 


255 


72.6 


80.7 


220 


61.8 


69.6 


Native 


Other 


365 


74.4 


84.1 


312 


61.8 


71.9 


Unknown race/ethnicity 


22 


47.6 


56.4 


13 


32.9 


33.3 


Year of birth 


1992 


4,152 


77.6 


84.4 


3,674 


68.3 


74.7 


1993 


10,041 


77.7 


84.9 


8,624 


66.7 


72.9 


Other/unknown 


87 


59.5 


72.5 


61 


43.9 


50.8 



^ Based on ECLS-K survey data and not on data from the sampling frame. 

^ A completed questionnaire was defined as one that was not completely left blank. 

^ The completion of the school-level instruments does not depend on whether the child has completed assessment or parent interview data. Hence, while all children with completed 
assessment and parent interview data have known value of gender; there is one child with completed facilities checklist data whose gender is unknown. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 
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Table 6-12. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, by child 
characteristic — children sampled in the base year 





Teacher questionnaire part A 


Teacher questionnaire part B 


Teacher questionnaire part C 


Child characteristics* 




Completion rates 






Completion rates 






Completion rates 






Completes^ 


Weighted Unweighted 


Completes^ 


Weighted Unweighted 


Completes^ 


Weighted Unweighted 


All students 


11,770 


62.4 


69.8 


11,741 


62.3 


69.6 


11,802 


62.7 


70.0 


Sex^ 


Male 


5,930 


61.4 


69.0 


5,921 


61.4 


68.9 


5,951 


61.7 


69.3 


Female 


5,840 


63.7 


70.7 


5,820 


63.4 


70.4 


5,851 


63.9 


70.8 


Race/ethnicity 


White (not Flispanic) 


7,174 


68.6 


76.8 


7,158 


68.4 


76.6 


7,186 


68.8 


76.9 


Black (not Flispanic) 


1,405 


53.7 


61.0 


1,402 


53.6 


60.8 


1,409 


54.0 


61.1 


Flispanic 


1,819 


54.7 


58.7 


1,820 


54.7 


58.7 


1,837 


55.2 


59.3 


Asian 


726 


59.8 


64.2 


714 


58.7 


63.2 


720 


59.1 


63.7 


Pacific Islander 


158 


74.2 


79.8 


157 


73.8 


79.3 


157 


73.8 


79.3 


American Indian or Alaska 


189 


53.3 


59.8 


189 


53.3 


59.8 


191 


53.1 


60.4 


Native 


Other 


288 


54.3 


66.4 


289 


55.0 


66.6 


289 


55.1 


66.6 


Unknown race/ethnicity 


11 


21.8 


28.2 


12 


27.7 


30.8 


13 


29.0 


33.3 


Year of birth 


1992 


3,491 


63.3 


71.0 


3,488 


63.2 


70.9 


3,497 


63.5 


71.1 


1993 


8,215 


62.3 


69.4 


8,188 


62.2 


69.2 


8,241 


62.6 


69.7 


Other/unknown 


64 


38.6 


53.3 


65 


40.2 


54.2 


64 


39.6 


53.3 



* Based on ECLS-K survey data and not on data from the sampling frame. 

^ A completed questionnaire was defined as one that was not completely left blank. 

^ There is a small number of children whose gender is unknown and who did not have completed teacher questionnaire data. The completion rates for these children, being zero, are not 
included in the table. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 



Data were also collected during spring-third grade from the special education teachers for 
children in special education programs (fewer than 1,200). Because this sample of children is small, 
completion rates were not computed for the different subgroups. Table 6-13 presents counts of completes 
and weighted and unweighted completion rates at the overall student level for the special education 
teacher questionnaires A and B. The completion rates for these instruments are higher than for the regular 
teacher questionnaires, 73.0 percent for part A, which captures teacher information, and 72.8 percent for 
part B, which relates to individual students who receive special education services. 



Table 6-13. ECLS-K, spring-third grade (2002): Number of completed instruments and child-level 

completion rates for special education teacher questionnaires — children sampled in the base 
year 



Category 




Completion rates 




Completes 


Weighted 


Unweighted 


Special education part A' 


875 


73.0 


75.0 


Special education part B‘ 


870 


72.8 


74.6 



’ A completed instrument was defined as one that was not completely left blank. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



6.2.2 Children Sampled in First Grade 

In spring-first grade, the student sample was freshened to include first graders who had no 
chance of selection in the base year because they did not attend kindergarten in the United States or were 
in first grade in the fall of 1998. For a detailed description of the freshening procedure see chapter 3, 
section 3.4.2. This same sample of students was followed into spring-third grade. Nonresponse in the 
freshened student sample could occur at two stages: during the procedure for sampling schools for 
freshening and identifying children to be used as freshening links in spring-first grade (first component) 
and then during data collection from the freshened students in spring-third grade (second component). 

The first component of the completion rate is the proportion of students sampled from the 
base year and subsampled for freshening that the study was able to do freshening on. The numerator 
includes all children who could be freshened from (i.e., those who did not move and in schools that 
cooperated with the freshening in first grade); the denominator includes children sampled for freshening 
(excluding movers not subsampled). For the weighted first component of the completion rate, both 
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numerator and denominator were weighted by the product of the school base weight, the within-school 
child weight, and the freshening adjustment factor. The school base weight and the within-school child 
level weight reflect the multi-stage sampling of the ECLS-K design, while the freshening adjustment 
factor is necessary because schools were subsampled for freshening in first grade as described above. 
These weights and adjustment factor are discussed in more detail in chapter 7. The first component alone 
can further be decomposed into two sources: attrition due to entire schools refusing to implement the 
freshening procedure (the school term), and attrition due to ECLS-K sampled children moving to other 
schools (the child term). To contain costs, students who transferred from schools targeted for freshening 
were not used as links to identify freshened students, even when they were otherwise followed for data 
collection. These movers were considered freshening nonrespondents in the child term. 

The second component is the proportion of freshened children with completed data for the 
instrument from within the population brought into the sample by freshening. The weight for this 
component is the product of the school base weight, the within-school child weight, the school freshening 
subsample adjustment factor and the third grade mover subsampling adjustment factor (which is equal to 
1 for children sampled in first grade since none of them were subsampled out for followup). 



The final completion rate is the product of the two components. For example, the final 
completion rate for child assessment is computed as follows: 



rcA 



XWy Z ^2i 

_ ieA 

Z ^2i 



i&B i kjENRca 



where A is the set of children who could be freshened from as described above, B is the set of children 
sampled for freshening, Wji is the weight for child i for the first component as described above, W 21 is the 
weight for child i for the second component as described above, ERca denotes eligible child assessment 
respondent, and ENRca eligible child assessment nonrespondent. To compute the unweighted rates, Wn 
and W 2 u are set to 1 for each child. 

Table 6-14 presents weighted and unweighted completion rates for freshened students. The 
two components of the completion rates are presented separately in table 6-14. The actual completion 
rates (i.e., the third set of rates in the table) are the products of the two components. Since no freshening 
was done in the third grade, the first component of the third-grade completion rate for freshened students 
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is identical to the first component of the spring-first grade completion rate for this same group of students. 
It is separated into a school term and a child term as described earlier. For this component, the completion 
rate is defined as the freshening completion rates, as opposed to the survey instrument completion rates 
found in the second component. The weighted freshening completion rate for children in schools targeted 
for freshening {the school term) is 65.4 percent. As part of the freshening process, schools were asked to 
prepare an alphabetic roster of students enrolled in first grade. These schools were also requested to 
identify which students did not attend kindergarten the previous year. Schools did not participate in the 
freshening process because they either refused or were unable to provide the requested information. 
Within the schools that agreed to freshen, the freshening completion rate is 98.3 percent, the slight loss 
due to students who transferred to other schools {the child term). Multiplying these two terms together 
gives a first component completion rate of 64.3 percent. Note that the first component rate for spring-third 
grade is identical to the first component rate for spring- first grade. 

The second component varies by survey instrument. The rates for the paper-and-pencil 
instruments range from 50.9 percent for the child-level teacher questionnaire to 80.3 percent for the 
facilities checklist and are uniformly lower than for the kindergarten sample. The child assessment at 78.3 
percent is 3 points lower than for the kindergarten sample, and the parent interview at 63.7 percent is 14 
points lower. The larger difference in the parent interview rates, as compared with the difference in the 
rates for child assessment, is due to the different number of followups the two groups of parents had. For 
parents of children sampled in first grade, the third grade data collection was the first followup while most 
parents of children sampled in kindergarten had at least three followups (spring-kindergarten, spring-first 
grade and spring-third grade). The large parent nonresponse rate is consistent with what is found in most 
longitudinal studies where the biggest sample attrition is in the first followup. The number of followups 
does not affect the completion rate of child assessment in the same way since child assessment was done 
in the school for most children and hence could be perceived as part of school activities. The final 
completion rate for each instrument is the product of the two components. Because of the low rates at the 
first stage, these range from a high of 51.7 percent for the facilities checklist to a low of 32.7 percent for 
the child-level teacher questionnaire. 



6-28 




Table 6-14. ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level 
completion rates for children sampled in first grade 



Category 




Completion rates' 


Completes 


Weighted 


Unweighted 


First component 


7,135 


64.3 


77.9 


School term^ 


7,192 


65.4 


78.5 


Child term^ 


7,135 


98.3 


99.2 


Second component 


Child assessments"* 


121 


78.3 


76.1 


Parent interview^ 


97 


63.7 


61.0 


Teacher questionnaire part A® 


86 


53.1 


54.1 


Teacher questionnaire part 


85 


52.2 


53.5 


Teacher questionnaire part C® 


82 


50.9 


51.6 


Special education part A*’ 


12 


67.6 


70.6 


Special education part B® 


13 


73.3 


76.5 


School administrator questionnaire^ 


102 


65.8 


64.2 


School fact sheet® 


124 


78.9 


78.0 


Facilities check list® 


126 


80.3 


79.2 


Student records abstract® 


103 


68.7 


64.8 


Overall completion rates 


Child assessments"^ 


121 


50.4 


59.3 


Parent interview® 


97 


41.0 


47.5 


Teacher questionnaire part A® 


86 


34.2 


42.1 


Teacher questionnaire part B® 


85 


33.6 


41.6 


Teacher questionnaire part C® 


82 


32.7 


40.2 


Special education part A® 


12 


43.5 


55.0 


Special education part B® 


13 


47.1 


59.6 


School administrator questionnaire® 


102 


42.3 


50.0 


School fact sheet® 


124 


50.8 


60.7 


Facilities check list® 


126 


51.7 


61.7 


Student records abstract® 


103 


44.2 


50.5 



' In the first component, this is the completion rate for freshening. In the second component, this is the completion rate for the survey instruments. 
The product of the two components is the overall completion rate for the survey instruments. 

^ The freshening completes and completion rates for children in schools targeted for freshening. 

^ The freshening completes and completion rates for children in schools that agreed to the freshening procedure. 

Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

^ A completed questionnaire was defined as one that was not completely left blank. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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6.2.3 



Spring-Third Grade Completion Rates for All Children 



To compute the completion rate for the combined set of children sampled in the base year 
and children sampled in first grade, the completion rate for each group is weighted by the proportion of 
all children in that group, and the two weighted completion rates were added together. For example, the 
weighted completion rate for the child assessment (CA) was computed as 



^CA - fCA,BY ^ 



Z 

+ ^CA 1ST ^ 



z 

i^lST 

z 



i^BYulST ieBY'ulST 



where BY denotes base year, 1ST denotes first grade, ycam is the child assessment completion rate for 
children sampled in the base year, vca.ist is the child assessment completion rate for children sampled in 
first grade, and Wi is the final weight (C5CW0 for child assessment) for child i. 

To get the weighted completion rate for the child assessment (which is 80.8 percent for 
children sampled in the base year and 50.4 percent for children sampled in first grade), the weighted 
proportion of children who were sampled in the base year was 0.9766; the weighted proportion of 
children who were sampled in first grade was 0.0234. The weighted completion rate for the child 
assessment was 0.808x0.9766+0.504x0.0234=0.801, or 80.1 percent. 

Table 6-15 presents final spring-third grade completion rates for children sampled in 
kindergarten, children sampled in first grade, and all children combined. Because children sampled in first 
grade represent such a small fraction of the total population of children, their inclusion in the computation 
of the completion rate brings down the rates for all children by less than one percent relative to the rates 
for children sampled in kindergarten, even though the completion rates for children sampled in first grade 
are lower than the kindergarten rates. 

Table 6-16 shows the completion rates for the child assessments, the parent interviews, and 
the school and teacher instruments for children who have nonzero child weights (C5CW0>0). These are 
children whose spring-third grade reading, math, or science assessments were scorable, or children who 
could not be assessed because of disabilities. These conditioned completion rates are useful to analysts 
who want to assess the relationship between the different instruments in term of participation. The 
completion rates from the different instruments are dependent in that if data from one instrument are 
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missing (e.g., parent instrument) it is likely that data from other instruments are also missing, (e.g., school 
administrator questionnaire). The conditioned completion rate for the child assessments are by definition 
100 percent. The less than 100 percent rate shown when children sampled in kindergarten are combined 
with children sampled in first grade is due to the school freshening nonresponse for children sampled in 
first grade. 



When the completion rates are conditioned on the presence of the child assessment data, they 
are at least 9 points higher than the unconditioned completion rates for all instruments except for the two 
special education questionnaires. For these last two instruments, the difference between the number of 
completes for the conditioned and unconditioned rates is very small; hence the conditioned rates are not 
affected as much as for the other instruments. For all the other instruments, the conditioned completion 
rates are higher by 9 points for the parent interviews and as high as 17 points for the student records 
abstract. 



Since data were collected from schools, parents, teachers, and children, there were many 
opportunities for sources to contribute differentially to nonresponse, and this is reflected in the varying 
completion rates. These completion rates differ not only by survey instruments, but within each survey 
instrument they are also different by school and child characters. A separate report examines the potential 
for bias in estimates produced from the ECLS-K third grade data. Since analysis of the third grade data is 
conditioned on the base year — only base year respondents were included in the collection of first grade 
and third grade data — the analysis of nonresponse bias is built on the base year nonresponse bias analysis 
(see the working paper Analysis of Nonresponse Bias in the Base Year Early Childhood Longitudinal 
Survey, Kindergarten Class of 1998-99 [U.S. Department of Education, National Center for Education 
Statistics, forthcoming). 
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Table 6-15, ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, for children sampled in 
kindergarten and first grade, by survey instruments 





Children sampled in kindergarten 


Children sampled in 


first grade 




All children 


Survey instrument 




Completion rates 




Completion rates 




Completion rates 




Completes 


Weighted 


Unweighted 


Completes 


Weighted 


Unweighted 


Completes 


Weighted 


Unweighted 


Child assessment* 


14,349 


80.8 


86.1 


121 


50.4 


59.3 


14,470 


80.1 


85.9 


Parent interview^ 


13,392 


77.8 


80.3 


97 


41.0 


47.5 


13,489 


76.9 


80.1 


Teacher questionnaire part 


11,770 


62.4 


69.8 


86 


34.2 


42.1 


11,856 


61.7 


69.6 


Teacher questionnaire part 


11,741 


62.3 


69.6 


85 


33.6 


41.6 


11,826 


61.6 


69.4 


Teacher questionnaire part 


11,802 


62.7 


70.0 


82 


32.7 


40.2 


11,884 


62.0 


69.7 


Special education part 


875 


73.0 


75.0 


12 


43.5 


55.0 


887 


72.3 


74.8 


Special education part 


870 


72.8 


74.6 


13 


47.1 


59.6 


883 


72.2 


74.5 


School administrator 


12,361 


66.1 


73.3 


102 


42.3 


50.0 


12,463 


65.5 


73.1 


questionnaire^ 
School fact sheet^ 


14,064 


76.5 


83.4 


124 


50.8 


60.7 


14,188 


75.9 


83.2 


Facilities check list^ 


14,280 


77.5 


84.7 


126 


51.7 


61.7 


14,406 


76.9 


84.5 


Student records abstract^ 


12,359 


67.0 


73.3 


103 


44.2 


50.5 


12,462 


66.5 


73.1 



* Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

^ A completed questionnaire was defined as one that was not completely left blank. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 
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Table 6-16, ECLS-K, spring-third grade (2002): Number of completed child-level cases and child-level completion rates, for children with 
scorable reading, math or science assessment or children not assessed due to disabilities, by survey instruments 





Children sampled in kindergarten 


Children sampled in 


first grade 




All children 


Survey instrument 




Completion rates 




Completion rates 




Completion rates 




Completes 


Weighted 


Unweighted 


Completes 


Weighted 


Unweighted 


Completes 


Weighted 


Unweighted 


Child assessment* 


14,349 


100.0 


100.0 


121 


64.3 


77.9 


14,470 


99.2 


99.8 


Parent interview^ 


12,564 


86.8 


87.6 


90 


49.7 


57.9 


12,654 


85.9 


87.4 


Teacher questionnaire part 


11,644 


78.0 


81.4 


86 


43.7 


55.4 


11,730 


77.2 


81.2 


Teacher questionnaire part 


11,614 


77.8 


81.2 


85 


43.0 


54.7 


11,699 


77.0 


81.0 


Teacher questionnaire part 


11,684 


78.4 


81.7 


82 


41.9 


52.8 


11,766 


77.5 


81.5 


Special education part 


854 


74.1 


75.8 


12 


43.5 


55.0 


866 


73.4 


75.6 


Special education part 


847 


73.7 


75.2 


13 


47.1 


59.6 


860 


73.1 


75.1 


School administrator 


11,846 


80.1 


82.8 


93 


50.6 


59.9 


11,939 


79.4 


82.6 


questionnaire^ 
School fact sheet^ 


13,447 


92.5 


94.0 


113 


60.4 


72.7 


13,560 


91.7 


93.8 


Facilities check list^ 


13,670 


94.0 


95.5 


116 


62.0 


74.7 


13,786 


93.3 


95.3 


Student records abstract^ 


12,282 


84.2 


85.8 


103 


56.5 


66.3 


12,385 


83.6 


85.6 



' Reading, math, or science assessment was scorable, or child was disabled and could not be assessed. 

^ Family structure portion of parent interview was completed. 

^ A completed questionnaire was defined as one that was not completely left blank. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, 
school year 2001-02. 



6.3 



Item Response Rates 



In the ECLS-K, as in most surveys, the responses to some data items are not obtained for all 
interviews. There are numerous reasons for item nonresponse. Some respondents do not know the answer 
for the item or do not wish to respond for other reasons. Some item nonresponse arises when an interview 
is interrupted and not continued later, leaving items at the end of the interview blank. Item nonresponse 
may also be encountered because responses provided by the respondent are not internally consistent, and 
this inconsistency is not discovered until after the interview is completed. In these cases, the items that 
were not internally consistent were set to missing. 

Every item in the ECES-K data fde has values that indicate whether the respondent did not 
know the answer to the item (-8), or refused to give an answer (-7). The value -9 is used in all other cases 
where the answer is left blank or set to missing due to reasons mentioned above (described in the data file 
as “Not ascertained”). However, where an item is left blank due to a valid skip pattern, this is indicated by 
the value -1. Chapter 7 of the ECLS-K User’s Manual for the Third Grade Public-Use Data File and 
Electronic Code Book. (NCES 2004-001) discusses in detail these special values. For each survey item, 
the response rate was computed as the number of responses not equal to any of the special values (- 1,-7,- 
8, or -9) divided by the number of responses not equal to - 1 . Of all the ECES-K instruments, only the 
parent interview has sizable number of items with special values -7 (“Refused”) or -8 (“Don’t know”). 
Table 6-17 shows the distribution of the nonresponse values for each instrument. 
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Table 6-17. ECLS-K, spring-third grade (2002): Number of survey items and percent of nonresponse 
values 



Survey instrument 




Percent of items with nonresponse value 




Number of 
items 


-7 

(Refused) 


-8 -9 

(Don’t know) (Not ascertained) 


Total 


1,953 


21 


27 


80 


Child assessment data 


62 


7 


8 


90 


Parent interview 


873 


44 


58 


58 


Teacher questionnaire part A 


319 


1 


1 


98 


Teacher questionnaire part B 


89 


3 


0 


99 


Teacher questionnaire part C 


99 


0 


0 


99 


Special education part A 


53 


0 


0 


98 


Special education part B 


106 


0 


0 


98 


School administrator questionnaire 


223 


3 


5 


99 


School fact sheet 


45 


0 


0 


100 


Facilities checklist 


43 


0 


0 


98 


Student records abstract 


41 


0 


0 


95 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 third grade data collection, school year 2001-02. 



For most of the data items collected in the ECLS-K, the item response rate was very high. 
Overall, the median item response rate is 98.3 percent. The median item response rate for each of the 
instruments ranges from 87.8 for the school fact sheet to 99.7 for the child assessment. Table 6-18 shows 
the number of items, the median item response rate, the lowest item response rate, the highest item 
response rate, and the number of items with response rates of less than 85 percent for all instruments. 
Items with less than 85 percent response rates are listed in table 6-19 by instrument and in ascending 
order of item response rate. The number of cases for which each item was attempted is also shown in this 
table. The tables in this chapter show the item response rates for items on the restricted-use fde. 
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Table 6- 1 8. ECLS-K, spring-third grade (2002): Item response rates for items on the ECLS-K 
third grade restricted-use data fde 



Instrument 


Number of 

Median Lowest Highest items with 

Number of response rate response rate response rate response rate 
items (percent) (percent) (percent) < 85 percent 


Total 


1,953 


98.3 


19.0 


100 


186 


Child assessment data 


62 


99.7 


92.2 


100 


0 


Parent interview 


873 


99.1 


19.0 


100 


72 


Teaeher questionnaire part A 


319 


98.2 


19.0 


100 


40 


Teacher questionnaire part B 


89 


98.7 


87.6 


100 


0 


Teacher questionnaire part C 


99 


98.7 


81.9 


100 


2 


Special education part A 


53 


96.2 


88.9 


100 


0 


Special education part B 


106 


97.4 


81.6 


100 


2 


School administrator questionnaire 


223 


96.3 


32.5 


100 


45 


School fact sheet 


45 


87.8 


58.1 


99.6 


11 


Facilities checklist 


43 


95.8 


88.8 


100 


0 


Student records abstract 


41 


92.5 


41.7 


100 


14 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten 
Class of 1998-99 third grade data collection, school year 2001-02. 
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Table 6-19. Items on the ECLS-K third grade (2002) restricted-use data fde with less than 85 percent 
response rates 



Variable name 


Description 


Number 

eligible 


Item response rate 
(percent) 


Parent Interview 








P5HEARS2 


P5 CHQ260 DEVICE EFFECT ON CHD’S HEARING 


95 


19.0 


P50FTCUT 


P5 FDQ230 FREQ CHILD SKIP MEAL-NO FOOD 


117 


21.4 


P5DGBEMM 


P5 CHQ340 MNTH AT 1ST DIAGNS-BEHAVIOR 


120 


28.3 


P5HEARS 


P5 CHQ230 DEGREE OF CHILD’S DEAFNESS 


127 


29.1 


P5DGBEYY 


P5 CHQ345 YR AT 1ST DIAGNOSIS-BEHAVIOR 


113 


30.1 


P5MMDIA5 


P5 CHQ220 MONTH AT 1ST DIAGNOSIS-HEARING 


127 


30.7 


P5DGNBEH 


P5 CHQ337 1ST DIAGNOSIS-BEHAVIOR 


120 


34.2 


P5YYDIA5 


P5 CHQ225 YR AT 1ST DIAGNOSIS-HEARING 


127 


37.0 


P5DIABEH 


P5 CHQ335 BEHAVIOR PROBLEM DIAGNOSED 


127 


37.8 


P5HEARAI 


P5 CHQ240 IF CHILD WEARS HEARING AID 


127 


39.4 


P5COCHLE 


P5 CHQ250 IF CHILD HAS COCHLEAR IMPLANTS 


127 


39.4 


P5NOTEA2 


P5 FDQ200 FREQ NOT EAT ENTIRE DAY 


170 


45.9 


P5MMDIA2 


P5 CHQ130 MNTH AT 1ST DIAGNOSIS-ACTIVITY 


180 


47.2 


P5CHIN3 


P5 NRQ050 CONTACT W/BIODAD SAME AS CHDl 


66 


50.0 


P5LRSRVM 


P5 CHQ530 MNTH LAST RECEIVED SERVICES 


182 


51.1 


P5EVBEH 


P5 CHQ330 CHD BEHAVIOR EVALUATED 


168 


52.4 


P5DIFFH3 


P5 CHQ210 IF HEAR DIFFICULTY DIAGNOSED 


166 


53.6 


P5CHROTH 


P5 CCQ403 # HRS/WK AT OTHER PROGRAMS 


117 


55.6 


P5LRSRVY 


P5 CHQ535 YR LAST RECEIVED SERVICES 


182 


56.0 


P5YYDIA2 


P5 CHQ135 YR AT 1ST DIAGNOSIS-ACTIVITY 


180 


56.7 


P5DGNACT 


P5 CHQ120 WHAT 1ST DIAGNOSIS - ACTIVITY 


180 


57.8 


P5CONBEH 


P5 CHQ327 CONCERNS ABOUT BEHAVIOR 


193 


59.1 


P5NHROTH 


P5 CCQ250 # HRS/WK RECVS CARE OTHER NREL 


134 


61.2 


P5DIFFH2 


P5 CHQ200 IF HEAR DIFFICULTY EVALUATED 


207 


61.4 


P5PROFFD 


P5 CHQl 10 IF ACTIVITY PROBLEM DIAGNOSED 


217 


65.0 


P5MMDIA4 


P5 CHQl 80 MONTH AT 1ST DIAGNOSIS-SPEECH 


431 


67.8 


P5LKSA 


P5 WPQ106 LOOK FOR WORK - SA 


311 


67.9 


P5PDJBSA 


P5 WPQ106 WORK FOR PAY - SA 


311 


67.9 


P5UNJBSA 


P5 WPQ106 WORK FOR NO PAY - SA 


311 


67.9 


P5SCHSA 


P5 WPQ106 ATTEND SCHOOL -SA 


311 


67.9 


P50THSA 


P5 WPQ106 OTHER - SA 


311 


67.9 


P5DGEMMM 


P5 CHQ370 MNTH AT 1ST DIAGNS-EMOTION BEH 


441 


70.3 


P5TAK2 


P5 EMQIOO PERS 2 JOB AVAILABLE LAST WEEK 


296 


72.0 


P5D01_2 


P5 EMQ070 PERS 2 CHKD W/PUB EMPL AGNCY 


307 


72.3 


P5D022 


P5 EMQ070 PERS 2 CHKD W/PRIV BMP AGNCY 


307 


72.3 


P5D03_2 


P5 EMQ070 PERS 2 CHKD W/EMPLOYR DIRECTLY 


307 


72.3 


P5D042 


P5 EMQ070 PERS 2 CHKD W/FRIENDS & REL 


307 


72.3 


P5D052 


P5 EMQ070 PERS 2 PLACED OR ANSWERED ADS 


307 


72.3 


P5D062 


P5 EMQ070 PERS 2 READ WANT ADS 


307 


72.3 


P5D07_2 


P5 EMQ070 PERS 2 DID SOMETHING ELSE 


307 


72.3 


P5DOW2 


P5 EMQ080 WHAT PERSON 2 DOING LAST WEEK 


762 


75.7 


P5LKFS 


P5 WPQ106 LOOK FOR WORK - FS 


425 


76.2 


P5PDJBFS 


P5 WPQ106 WORK FOR PAY - FS 


425 


76.2 


P5UNJBFS 


P5 WPQ106 WORK FOR NO PAY - FS 


425 


76.2 



See note at end of table. 
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Table 6-19. Items on the ECLS-K third grade (2002) restricted-use data fde with less than 85 percent 
response rates — Continued 



Variable name 


Description 


Number 

eligible 


Item response rate 
(percent) 


P5SCHFS 


P5 WPQ106 ATTEND SCHOOL -FS 


425 


76.2 


P50THFS 


P5 WPQ106 OTHER -FS 


425 


76.2 


P5AGREE3 


P5 NRQ264 AGREEMENT W/ ADOPTIVE FATHER 


148 


77.7 


P5BESTEY 


P5 CHQ320 WHAT CAN CHILD BEST SEE 


475 


77.9 


P5MMDIAG 


P5 CHQ070 MNTH AT 1ST DIAGNS-LRN ABLTY 


959 


78.2 


P5LOK2 


P5 EMQ060 PERS 2 SOUGHT JOB LAST 4 WEEKS 


842 


78.3 


P5RESPON 


P5 CHQIOO IF ACTIVITY LEVEL EVALUATED 


353 


78.8 


P5TAK1 


P5 EMQIOO PERS 1 JOB AVAILABLE LAST WEEK 


526 


78.9 


P5DGEMYY 


P5 CHQ375 YR AT 1ST DIAGNS-EMOTIONAL BEH 


441 


78.9 


P5DGNEMO 


P5 CHQ365 1ST DIAGNOSIS-EMOTIONAL BEH 


438 


79.7 


P5VAC_2 


P5 EMQ030 IF PERS 2 ON LEAVE PAST WEEK 


912 


80.2 


P5D01_1 


P5 EMQ070 PERS 1 CHKD W/PUB EMPL AGNCY 


565 


80.2 


P5D021 


P5 EMQ070 PERS 1 CHKD W/PRIV BMP AGNCY 


565 


80.2 


P5D03_1 


P5 EMQ070 PERS 1 CHKD W/EMPLOYR DIRECTLY 


565 


80.2 


P5D041 


P5 EMQ070 PERS 1 CHKD W/FRIENDS & REL 


565 


80.2 


P5D051 


P5 EMQ070 PERS 1 PLACED OR ANSWERED ADS 


565 


80.2 


P5D061 


P5 EMQ070 PERS 1 READ WANT ADS 


565 


80.2 


P5D07_1 


P5 EMQ070 PERS 1 DID SOMETHING ELSE 


565 


80.2 


P5YYDIA4 


P5 CHQ185 YEAR AT 1ST DIAGNOSIS-SPEECH 


431 


80.5 


P5CHIN1 


P5 NRQ050 CONTACT W/BIOMOM SAME AS CHDl 


11 


81.8 


P5TWNCAR 


P5 CCQ003 TWIN SAME CARE AS CHILD 


308 


82.5 


P5EVCUT 


P5 FDQ150 FREQ CUT MEAL SIZE 


528 


82.6 


P5CCNMCH 


P5 CCQ400 CC # CHILDREN AMOUNT FOR 


304 


83.2 


P5REL15 


P5 ROSTER 15TH PERSON’S RELATION TYPE 


6 


83.3 


P5RCNMCH 


P5 CCQ135 RC # CHILDREN AMOUNT FOR 


289 


83.4 


P5REQSA 


P5 WPQ105 ANY RQ FOR STATE AID (SA) 


671 


84.1 


P5RCCOST 


P5 CCQ125 COST OF RELATIVE CARE 


452 


84.3 


P5COMMU2 


P5 CHQ170 IF SPEECH PROBLEM DIAGNOSED 


516 


84.7 


Teacher questionnaire part A 






A50THTST 


A5 Q60H OTHER ASSESSMENTS 


11,781 


19.0 


A50TASSI 


A5 Q29E FREQUENCY OTHER SPECIFY HELP 


11,781 


24.0 


A5ESLXIN 


A5 Q38B3 # ESL PAID AIDE DO OTHER TASKS 


5,289 


39.1 


A5ESLINS 


A5 Q38A3 # ESL PAID AIDE WORK W/CHLDREN 


5,289 


41.3 


A5SPEXIN 


A5 Q38B2 # SPED PAID AIDE DO OTHER TASKS 


5,289 


41.5 


A5SPEINS 


A5 Q38A2 # SPED PAID AIDE WORK W/CHLDREN 


5,289 


50.4 


A5NCDIS 


A5 Q120 NOT CLASSIFIED DISABILITIES 


7,205 


56.8 


A50TDIS 


A5 Q12N OTHER SPECIFY DISABILITIES 


7,205 


64.0 


A5ELRPBR 


A5 Q6B ELIGIBLE REDUCED-PRICE BREAKFAST 


11,781 


65.9 


A5NOESL 


A5 Q18A LEP STUDENTS GET NO ESL 


2,616 


68.1 


A5ELRPLU 


A5 Q6D ELIGIBLE REDUCED-PRICE LUNCH 


11,781 


69.5 


A5DEAF 


A5 Q12M DEAFNESS AND BLINDNESS 


7,205 


73.5 


A5TRAUM 


A5 Q12L TRAUMATIC BRAIN INJURIES 


7,205 


73.7 


A5MULTI 


A5 Q12J MULTIPLE DISABILITIES 


7,205 


73.9 


A5DELAY 


A5 Q12E DEVELOPMENTAL DELAY 


7,205 


74.0 



See note at end of table. 
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Table 6-19. Items on the ECLS-K third grade (2002) restricted-use data fde with less than 85 percent 
response rates — Continued 



Variable name 


Description 


Number 

eligible 


Item response rate 
(percent) 


A50RTH0 


A5 Q12H ORTHOPEDIC IMPAIRMENTS 


7,205 


74.1 


A5ELFSBR 


A5 Q6A ELIGIBLE FREE SCHOOL BREAKFAST 


11,781 


74.1 


A5RETAR 


A5 Q12D MENTAL RETARDATION 


7,205 


74.8 


A5AUTSM 


A5 Q12K AUTISM 


7,205 


75.1 


A50THER 


A5 Q12I OTHER HEALTH IMPAIRMENTS 


7,205 


75.1 


A5VIS 


A5 Q12F VISION IMPAIRMENTS 


7,205 


75.1 


A5HEAR 


A5 Q12G HEARING IMPAIRMENTS 


7,205 


75.4 


A5EMPRB 


A5 Q12C SERIOUS EMOTIONAL PROBLEMS 


7,205 


76.9 


A5ELFSLU 


A5 Q6C ELIGIBLE FREE SCHOOL LUNCH 


11,781 


77.1 


A5TXTHTR 


A5 Q26H TIME FOR THEATER 


5,018 


77.6 


A5ESLRE 


A5 Q18B LEP STUDENTS IN-CLASS ESL 


2,616 


79.2 


A5SC504 


A5 Q13C CHILDREN W/ SECTION 504 PLAN 


7,205 


80.0 


A5TXDAN 


A5 Q26G TIME FOR DANCE 


4,678 


81.0 


A5MNEXTR 


A5 Q29A MINUTES EXTRA ASSISTANCE 


10,828 


81.4 


A57YRSLS 


A5 Q3A # 7 YEARS OLD OR LESS IN CLASS 


11,781 


81.6 


A58YROL 


A5 Q3B HOW MANY 8-YEAR-OLDS IN CLASS 


11,781 


81.6 


A59YROL 


A5 Q3C HOW MANY 9-YEAR-OLDS IN CLASS 


11,781 


81.6 


A510YROL 


A5 Q3D HOW MANY lO-YEAR-OLDS IN CLASS 


11,781 


81.6 


A511YRMO 


A5 Q3E # 1 1 -YEAR OLDS & OLDER IN CLASS 


11,781 


81.6 


A5TOTAG 


A5 Q3F TOTAL CLASS ENROLLMENT (AGE) 


11,781 


81.6 


A5TSTPRP 


A5 Q64 HOURS SPENT IN STD TEST PREP 


10,760 


81.7 


A5TXREF 


A5 Q26K TIME FOR REFERENCE SKILLS 


10,926 


82.9 


A5ESLOU 


A5 Q18C LEP STUDENTS OUTSIDE ESL 


2,616 


83.0 


A5MNAIDE 


A5 Q29B MINUTES TUTORED BY AIDE 


7,223 


83.8 


A5REGXIN 


A5 Q38B1 # REG PAID AIDE DO OTHER TASKS 


5,289 


83.9 


Teacher questionnaire part C 






T5MACNS 


T5 Q2 CONSULT FAMILIAR PERSON 


897 


81.9 


T5SUMPRG 


T5 Q22 IF RETAINED ELIG FOR SUMMER PROG 


419 


84.7 


Special education part B 






E5EVLOTH 


E5 Q21G OTHER EVALUATION 


870 


81.6 


E50THSER 


E5 Q8J OTHER SERVICE PROVIDED 


870 


83.3 


School administrator questionnaire 






S50THTRA 


S5 Q36F OTHER LM-LEP SERVICES 


7,069 


32.5 


S5RTCHPT 


S5 Q40A2 # REG CLASSROOM TCHR-PART 


12,009 


48.1 


S5LIBRPT 


S5 Q40J2 # LIBRARIANS-PART 


12,009 


50.5 


S5PRABRK 


S5 Q12A2 PARTICIPATE ANY SCH BREAKFAST 


7,981 


52.0 


S5READPT 


S5 Q40E2 # READING TCHR/SPECIALIST-PART 


12,009 


54.3 


S5TEST12 


S5 Q26 TWELFTH GRADE TESTED 


11,817 


55.3 


S5ESLPT 


S5 Q40D2 # ESL/BILINGUAL TCHR-PART 


12,009 


55.4 


S5TEST11 


S5 Q26 ELEVENTH GRADE TESTED 


11,817 


55.5 


S5TEST10 


S5 Q26 TENTH GRADE TESTED 


11,817 


55.6 


S5TEST9 


S5 Q26 NINTH GRADE TESTED 


11,817 


55.8 



See note at end of table. 
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Table 6-19. Items on the ECLS-K third grade (2002) restricted-use data fde with less than 85 percent 
response rates — Continued 



Variable name 


Description 


Number 

eligible 


Item response rate 
(percent) 


S5SPEDPT 


S5 Q40C2 # SPECIAL ED TCHR-PART 


12,009 


55.8 


S5GIFTPT 


S5 Q40F2 # GIFTED/TALENTED TCHR-PART 


12,009 


58.6 


S5PARAPT 


S5 Q40I2 # PARAPROFESSIONALS-PART 


12,009 


59.1 


S50THER 


S5 Q8F SCH HAS OTHER REASON 


3,833 


59.7 


S5PSYCFL 


S5 Q40H1 SCH PSYCH/ SOCIAL WORKER-FULL 


12,009 


60.1 


S5NURSPT 


S5 Q40G2 # SCH NURSE HEALTH PROF-PART 


12,009 


62.7 


S5TEST8 


S5 Q26 EIGHTH GRADE TESTED 


11,817 


64.3 


S5ELRPBK 


S5 Q12C1 ELIGIBLE RED-PRICE BREAKFAST 


7,981 


64.4 


S5TEST7 


S5 Q26 SEVENTH GRADE TESTED 


11,817 


64.6 


S5ELIBRK 


S5 Q12B1 ELIGIBLE FOR FREE BREAKFAST 


7,981 


65.3 


S5MSARPT 


S5 Q40B2 # GYM DRAMA MUSIC ART TCHR-PART 


12,009 


67.9 


S5PARPBK 


S5 Q12C2 PARTICIPATE RED-PRICE BREAKFAST 


7,981 


68.0 


S5PAALUN 


S5 Q13A2 PARTICIPATE ANY SCH LUNCH 


12,009 


69.8 


S5NURSFL 


S5 Q40G1 # SCH NURSE HEALTH PROF-FULL 


12,009 


70.8 


S5GIFTFL 


S5 Q40F1 # GIFTED/TALENTED TCHR-FULL 


12,009 


71.0 


S5PSYCPT 


S5 Q40H2 SCH PSYCH/SOCIAL WORKER-PART 


12,009 


71.2 


S5TEST6 


S5 Q26 SIXTH GRADE TESTED 


11,817 


72.6 


S5HWLONG 


S5 Q42E RESP (NOT PRINCIPAL) YR AT SCH 


3,417 


73.0 


S5PARBRK 


S5 Q12B2 PARTICIPATES IN BREAKFAST 


7,981 


73.1 


S5BILING 


S5 Q35A BILINGUAL SERVICES PERCENT -3RD 


7,069 


73.4 


S5ESLBIL 


S5 Q35C ESL AND BILINGUAL PERCENT -3RD 


7,069 


73.4 


S5ESLFL 


S5 Q40D1 # ESL/BILINGUAL TCHR-FULL 


12,009 


73.5 


S5PCTMTH 


S5 Q27B MATHEMATICS SKILLS % 


11,817 


74.7 


S5PCTRD 


S5 Q27A READING OR VERBAL SKILLS % 


11,817 


75.5 


S5AMBUSL 


S5 Q5 TIME LAST BUS AM 


12,009 


78.8 


S5MSARFL 


S5 Q40B1 # GYM DRAMA MUSIC ART TCHR-FULL 


12,009 


79.5 


S5ELIRED 


S5 Q13C1 ELIGIBLE IN REDUCED-PRICE LUNCH 


12,009 


80.3 


S5READFL 


S5 Q40E1 # READING TCHR/SPECIALIST-FULL 


12,009 


81.0 


S5ELILNC 


S5 Q13B1 ELIGIBLE FOR FREE LUNCH 


12,009 


81.1 


S5ESLONL 


S5 Q35B ESL SERVICES PERCENT -3RD 


7,069 


81.2 


S5PARAFL 


S5 Q40H # PARAPROFESSIONALS-FULL 


12,009 


82.4 


S5AMBUSF 


S5 Q4 TIME FIRST BUS AM 


12,009 


83.0 


S5PARRED 


S5 Q13C2 PARTICIPATES IN RED-PRICE LUNCH 


12,009 


83.1 


S5LIBRFL 


S5 Q40J1 # LIBRARIANS-FULL 


12,009 


83.4 


S5PARLNC 


S5 Q13B2 PARTICIPATES IN FREE LUNCH 


12,009 


84.7 


School fact sheet 


L5NAISKL 


L5 Q4C PRIVATE-ACCREDITED BY NAIS 


3,927 


58.1 


L50THREL 


L5 Q4B PRIVATE, OTHER RELIGIOUS 


3,927 


58.4 


L5PRIVRD 


L5 Q4A4 IS IT A PRIVATE ORDER 


3,927 


58.5 


L5DIOCES 


L5 Q4A2 IS IT A DIOCESAN SCHOOL 


3,927 


58.6 


L5PARISH 


L5 Q4A3 IS IT A PARISH SCHOOL 


3,927 


58.6 


L50THPRI 


L5 Q4D IS IT OTHER PRIVATE 


3,927 


58.8 


L5PVTSPD 


L5 Q4E IS IT SPECIAL EDUCATION 


3,927 


59.1 


L5PVTEAR 


L5 Q4F IS IT AN EARLY CHILDHOOD CENTER 


3,927 


59.1 



See note at end of table. 
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Table 6-19. Items on the ECLS-K third grade (2002) restricted-use data fde with less than 85 percent 
response rates — Continued 



Variable name 


Number 


Item response rate 




Description 


eligible 


(percent) 


L5CATHOL 


L5 Q4A1 IS IT A CATHOLIC SCHOOL 


3,927 


59.6 


L5DD02 


L5 QlOB DAY COMPLETED 


13,641 


84.6 


L5MM02 


L5 QlOA MONTH COMPLETED 


13,641 


84.7 


Student records abstract 






U5WHYLFT 


U5 Q4 WHY CHILD LEFT SCHOOL 


300 


41.7 


U5DDLEFT 


U5 Q3B DAY LEFT SCHOOL 


300 


45.3 


U5MMLEFT 


U5 Q3A MONTH LEFT SCHOOL 


300 


45.7 


U5YYLEFT 


U5 Q3C YEAR LEFT SCHOOL 


300 


45.7 


U5YY00 


U5 Q9B3 YEAR 2000-01 lEP SIGNED 


1,076 


63.5 


U5MM00 


U5 Q9B2 MONTH 2000-01 lEP SIGNED 


1,076 


63.8 


U5IEP00 


U5 Q9B1 PRESENCE OF 2000-2001 lEP RECORD 


1,330 


76.5 


U5YY01 


U5 Q9A3 YEAR 2001-02 lEP SIGNED 


1,278 


76.8 


U5MM01 


U5 Q9A2 MONTH 2001-02 lEP SIGNED 


1,278 


77.8 


U5BUXTAR 


U5 Q7C CHILD UNEXCUSED TARDIES 


12,367 


80.3 


U5BXTARD 


U5 Q7B CHILD EXCUSED TARDIES 


12,367 


80.4 


U5AUXABS 


U5 Q6C CHILD UNEXCUSED ABSENCES 


12,367 


82.0 


U5AXABS 


U5 Q6B CHILD EXCUSED ABSENCES 


12,367 


82.2 


U5IEP01 


U5 Q9A1 PRESENCE OF 2001-2002 lEP RECORD 


1,330 


84.7 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 
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7. WEIGHTING AND VARIANCE ESTIMATION 



The ECLS-K data were weighted to compensate for differential probabilities of selection at 
each sampling stage and to adjust for the effects of nonresponse. As in the first grade year, only child- 
level weights were computed for third grade. The use of these weights was essential to produce estimates 
that are representative of the cohort of children who were in kindergarten in 1998-99 or in first grade in 
1999-2000. Since the third grade sample was not freshened with third graders who did not have a chance 
to be sampled in kindergarten or first grade (as was done in first grade), estimates from the ECLS-K third 
grade data are representative of the population cohort rather than all third graders in 2001-02. While the 
vast majority of children in third grade in the 2001-02 school year are members of the cohort, third 
graders who repeated second or third grade, children who were homeschooled before enrolling in third 
grade, and recent immigrants are not covered. Data were collected from teachers and schools to provide 
important contextual information about the school environment for the sampled children. Similarly, home 
environment data were collected from parents. Data from these sources are not representative of all third 
grade parents, teachers, and schools in 2001-02. For this reason, the only weights produced from the 
study are child-level weights for making statements about children, including statements about the 
parents, teachers, and schools of those children. 

The different types of weights are discussed in section 7.1, followed by a detailed 
description of the computation of the weights in section 7.2. Section 7.3 describes the variance estimation 
methods suitable for the ECLS-K. 



7.1 Types of Weights 

Two sets of weights were computed for third grade, cross-sectional and longitudinal. Cross- 
sectional weights were used to produce estimates that are representative of the cohort of children who 
were in kindergarten in 1998-99 or in first grade in 1999-2000.’ Longitudinal weights were used to 
analyze data in a longitudinal file created by merging base year, first grade, and third grade data. 



' Since the third grade sample was not freshened with third graders who did not have a chance to be sampled in kindergarten or first grade (as was 
done in first grade), estimates from the ECLS-K third grade data are representative of the population cohort rather than all third graders in 2001- 
02. The estimated number of children from the ECLS-K is approximately 96 percent of all third graders. While the vast majority of children in 
third grade in the 2001-02 school year are members of the cohort, third graders who repeated second or third grade and recent immigrants are not 
covered. 
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As in previous years, there were several survey instruments administered to sampled 
children and their parents, teachers, and schools: cognitive and physical assessments for children; parent 
interviews; several types of teacher and school questionnaires, and observation instruments (see the Early 
Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K) Base Year Public-Use Data 
Files and Electronic Code Book: User’s Manual (NCES 2001-029; U.S. Department of Education, 
National Center for Education Statistics, 2001) for a description of the instruments). The multiple stages 
of base year sampling in conjunction with differential nonresponse at each stage and the diversity of 
survey instruments require that multiple cross-sectional sampling weights be computed for use in 
analyzing the ECES-K data. Exhibit 7-1 summarizes the different types of cross-sectional weights. 



Exhibit 7-1. ECES-K third grade cross-sectional weights: School year 2001-02 



Weight to be used for analysis of ... 

C5CW0 third grade direct child assessment data, alone or in conjunction with any combination of 

(a) a limited set of child characteristics (e.g., age, sex, race/ethnicity), (b) any third grade 
teacher questionnaire A, B or C data, and (c) data from the school administrator 
questionnaire or school fact sheet. 

C5PW0 third grade parent interview data alone or in combination with (a) third grade child 

assessment data, (b) third grade teacher questionnaire A, B, or C data, and (c) data from 
the school administrator questionnaire or school fact sheet. 

Exception: If data from the parent AND child assessment AND teacher questionnaire A 
or B (not C) are used, then C5CPTW0 should be used. 



C5CPTW0 third grade direct child assessment data combined with third grade parent interview data 
AND third grade teacher data alone or in conjunction with data from the school 
administrator or school fact sheet or facilities checklist. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



The ECES-K longitudinal files are created by merging data from the base year, first grade, 
and third grade. Eongitudinal weights were created to use in analyzing data in these longitudinal files. 
These weights are described in exhibit 7-2. All longitudinal weights are child-level weights. 

As mentioned in the introduction, teachers and schools are not representative of third grade 
teachers and schools in 2001-02. For this reason, there are no cross-sectional weights computed to 
provide estimates at the school or teacher level. Consequently, there are no longitudinal weights 
computed at the school or teacher level. 
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Exhibit 7-2. ECLS-K: K-3 (panel) longitudinal weights: School year 2001-02 



Weights to be used for analysis of 



C45CW0 

C45PW0 

C245CW0 

C245PW0 

C1_5FC0 

C1_5FP0 

C1_5SC0 

Cl 5SP0 



direct child assessment data from BOTH spring-first grade and spring-third grade, alone 
or in conjunction with any combination of a limited set of child characteristics (e.g., age, 
sex, race/ethnicity). 

parent interview data from BOTH spring-first grade and spring-third grade. 

direct child assessment data from spring-kindergarten AND spring-first grade AND 
spring-third grade, alone or in conjunction with any combination of a limited set of child 
characteristics (e.g., age, sex, race/ethnicity). 

parent interview data from spring-kindergarten AND spring- first grade AND spring-third 
grade. 

direct child assessment data from four rounds of data collections involving the full 
sample of children (fall-kindergarten, spring-kindergarten, spring-first grade, spring-third 
grade), alone or in conjunction with any combination of a limited set of child 
characteristics (e.g., age, sex, race/ethnicity). 

parent interview data from four rounds of data collections involving the full sample of 
children (fall-kindergarten, spring-kindergarten, spring-first grade, spring-third grade). 

direct child assessment data from all five rounds of data collection, alone or in 
conjunction with any combination of a limited set of child characteristics (e.g., age, sex, 
and race/ethnicity). 

parent interview data from all five rounds of data collection. 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 third grade data collection, school year 2001-02. 



Each set of weights created to be used with the ECES-K data consists of a full sample weight 
that is used in computing survey estimates and replicate weights that are used in variance estimation with 
a jackknife replication method. First- stage stratum and primary sampling unit (PSU) identifiers are also 
created so that variance estimation using the Taylor series approximation method can be produced using 
the full sample weights. See section 7.2.5 for a description of how replicate weights were created. Section 
7.3 discusses variance estimation methods. 



The data file includes the final full sample weight (described in section 7.2.4) and the final 
replicate weights (described in section 7.2.5) but not the intermediate weights leading to the final weights. 
The names of the full sample weights in the file are as described in exhibits 7-1 and 7-2 (e.g., C5CW0). 
The names of the replicate weights have the same prefix as the full sample weight with the last digit 
indicating the replicate (e.g., C5CW1 to C5CW90 are the 90 replicate weights to be used with the full 
sample weight C5CW0). 
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7.2 



Computation of the Spring-Third Grade Weights 



The third grade sample included only base year respondents (see chapter 3) and a 
supplemental sample of first graders brought in through a sample freshening procedure implemented in 
spring-first grade. Only a subsample of children who moved from the schools they were attending when 
they were sampled originally was followed into their new schools. However, children who moved into a 
destination school^ because they had completed the highest grade at the originally sampled school were 
all followed. Also, children who belong to the language minority group and who had not moved out of the 
original sample schools at any time during the first grade year were all followed into their new third grade 
schools if they moved from the original sample school after spring-first grade. 

The weighting procedures for both cross-sectional and longitudinal weights are similar, 
although weighting cells were defined differently for each type of weight. For example, any longitudinal 
weight that contains data from fall-first grade may have used different cells due to sample size 
constraints. 



The weighting procedures for the third grade were divided into three main stages. These 
procedures were followed for creating each weight shown in exhibit 7-1 and exhibit 7-2. Only the groups 
of eligible children to whom the weight applies changed. For example, weight CSC WO pertains to 
children with completed assessments in spring-third grade; weight C45PW0 pertains to children with 
completed parent interview in both spring-first grade and spring-third grade. In the base year, children 
who were not assessed because of a disability or because they were language minority children had 
positive CICWO and C2CW0 weights because they had data such as age, gender, race/ethnicity, height 
and weight, and characteristics of parents, teachers, and classrooms. In subsequent rounds of data 
collection, they continued to be treated the same. 

Weights that include any fall-first grade data (such as C15SC0, which is the weight for 
children for whom child assessments were obtained in all five rounds) were computed using the same 
procedures, but the cells for the weighting adjustments were more restricted because only the fall-first 
grade subsample was included. The replication scheme for data that include the fall-first grade panel is 
also different as described in section 7.2.5. 



2 



A destination school is a school that received at least four students from the school where they had just completed the highest grade. 
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The first stage of weighting was to compute an initial child weight that reflects the 

following: 

■ Adjustment of the school base weight for base year school-level nonresponse; 

■ Adjustment of the base year child weight for base year child- level nonresponse; and 

■ Adjustment of the base year child weight for subsampling of schools for tfeshening in 

first grade (for children sampled in first grade only). 

The first-stage weights are exactly the same as those used for the first grade year. They were 
used in spring-third grade as the starting weights for the adjustments in the second stage. 

The second stage of weighting was to adjust the initial child weight computed in the first 
stage for the following: 

■ Subsampling of movers and 

■ Child-level nonresponse. 

The third and last stage was to rake the weights adjusted in the second stage to sample-based 
control totals. 

The procedures used in the four stages of weighting are described in more detail in the 
following sections of this chapter. 



7.2.1 Initial Child Weights 

As mentioned earlier, the first stage of weighting was to compute an initial child weight that 
reflects: (1) the adjustment of the school base weight for base year school-level nonresponse (school-level 
weights), (2) the adjustment of the base year child weight for base year child-level nonresponse (child- 
level weights), and (3) the adjustment of the base year child weight for subsampling of schools for 
freshening in first grade (child-level weights, for children sampled in first grade only). These weights 
were already computed for spring-first grade. For completeness, they are described below, in section 

7.2. 1 . 1 for the school-level weights and in section 7.2. 1 .2 for the child- level weights. 
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7.2.1.1 



Base Year Nonresponse-Adjusted School Weights 



The school base weight^ SCHLBWj was the same as that computed for previous rounds of 
data collection. It is the inverse of the probability of selection of the school from a stratified probability 
proportional to size (PPS) design: 



SCHLBWi = 



1 1 

POSpgu j POSgcHL i 



where 



POSpsu j is the probability of selection of the PSU j, and 
POSg(,fjp , is the probability of selection of school i within the PSU j 

where the probability of selection of the PSUy, POSp^p j , was defined as 



POS 



PSU j 



IV J 



if PSU j is a self-representing (SR) PSU, 
if PSU j is a non self-representing (NSR) PSU 



where 

M J is the measure of size (MOS) of PSU j (i.e., count of five-year-old children in the 

PSU as described in section 3.2.1), and 
M f, is the total MOS in stratum h, PSU j e h , 

and the probability of selection of school i within PSU j , POS^chl i > was defined as 



POSscHLi -^kj 



rtl: 



iGk j 



^ Only schools sampled in the base year have base weights. Transfer schools do not have base weights, but children in transfer schools carry with 
them the base weights of the original sampled schools. 
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where 



rij^j is the target number of schools in stratum k in PSU i, and 
w, is the MOS of the school i in stratum k in PSU i. 



For schools sampled using the new school sampling procedure/ the school base weight 
SCHLBWj was computed as 



SCHLBWi = 



1 

POSpsu I 



1 



X — X 

fi 



1 

POSgcHL i 



and the factor fi is defined as 



where 



fi- 



1 

P LEA 
P DIO 



if the school is a new non-Catholic private school, 
if the school is a new public school, 
if the school is a new Catholic school 



p is the within stratum selection probability of the school district, and 
p is the within stratum selection probability of the diocese. 

The school base weight was adjusted for base year school-level nonresponse. A base year 
responding school was an original sample school with at least one child with a positive CICWO, C2CW0, 
CIPWO, or C2PW0 weight. CICWO is positive for language minority (not Spanish) children, children 
with disabilities, and children with at least one direct cognitive test score in fall-kindergarten. CIPWO is 
positive for children whose parents completed the family structure questions of the parent interview in 
fall-kindergarten. C2CW0 and C2PW0 weights are positive under similar circumstances but for spring- 
kindergarten. Schools that did not meet this condition are nonrespondents and their weights were 
distributed across responding units (at the school level) in this stage. The base year school weight was 
adjusted within nonresponse weighting classes. The base year nonresponse-adjusted school weight 
SCHLADW^ was computed as 



SCHLADWi = SCHADF SCHLBWi 



The sample was expanded to account for newly opened schools and schools not found in the sampling frame. See chapter 3 for more details on 
how these schools were identified. 
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where 



SCHLBWj is the school base weight, and 

SCHADF^ is the base year school nonresponse adjustment factor for schools in cell c, 
calculated as 



^ SCHLBWi mii 

ieER^ ,ENR^ 



SCHADF^ = _ 

^ SCHLBWi xmi 

ieER. 



where 



FR^ denotes the set of eligible school respondents in cell c, 

ENR^ denotes the set of eligible school nonrespondents in cell c, and 

w, is the measure of size for school i (i.e., count of students in the school as described in 
section 3. 2.2.2) 

Base year school characteristics used for constructing nonresponse cells were the school type 
(public, Catholic private, non-Catholic religious private, or nonsectarian private), the school locale (large 
city, midsize city, suburb of large city, suburb of midsize city, large town, small town, or rural area), the 
region where the school is located (Northeast, Midwest, South, or West), and the size classification of the 
school in terms of school enrollment as described in table 7-1. 
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Table 7-1. Size classifieation for school nonresponse 
adjustment: School year 1998-99 



Size classification 


School enrollment 


Public 


Private 


1 


1-24 


1 - 11 


2 


25-39 


12-23 


3 


40-49 


24-35 


4 


50-59 


36-47 


5 


60-69 


48-59 


6 


70-79 


60 or more 


7 


80-89 




8 


90-99 




9 


100-119 




10 


120-139 




11 


140-179 




12 


180-219 




13 


220 or more 





SOURCE: U.S. Department of Education, National Center for Education 
Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 
1998-99 base year data collection, school year 1998-99. 



7.2, 1.2 Base Year Child Weights 

Two groups of children were fielded in spring-third grade: base year respondents, and 
eligible children who were sampled in first grade as part of the sample freshening procedure. The base 
year child weights for the two groups were the same as those computed for the first-grade year. A 
description of the two weights follows. 



7,2, 1.2,1 Base Year Child Weights for Base Year Respondents 

A base year respondent is a sampled child with a positive CICWO, C2CW0, Cl TWO, or 
C2PW0 weight. The CICWO weights are positive for language minority (not Spanish), disabled, and 
assessed children in fall-kindergarten. The CIPWO weights are positive for children whose parents 
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completed the family structure questions of the parent interview in fall-kindergarten. The C2CW0 and 
C2PW0 weights are positive under similar circumstances, but apply to data from spring-kindergarten. 

The base year child weight BYCHLDW^ was computed as 

BYCHLDWi = SCHLADWi x (VCHLD PROBi) 



where 



SCHLADWi is the base year nonresponse-adjusted school weight described in 
section 7. 2. 1.1, and 

CHLD PROBi is the probability of selection of the child within a school. 

To account for base year nonresponse — children who were not assessed in the base year and 
whose parent interviews were not completed (i.e., children who did not have at least one positive weight 
among CICWO, C2CW0, CIPWO and C2PW0) — the base year child weight was adjusted for 
nonresponse. The child weight adjusted for base year child-level nonresponse, ABYCHLDWi, was 

computed as 



ABYCHLDWi = BYADF^ x BYCHLDWi 



where BYADFc , the adjustment factor, was calculated as 



BYADF^ 



Y, BYCHLDWi 

i^BY R^ , BY_NR^ 

Y BYCHLDWi 

i^BY_R^ 



where 



BYRc denotes the set of base year child respondents in cell c, and 

BY NR c denotes the set of base year child nonrespondents in cell c. 

The base year child weights were adjusted using weighting classes similar to those 
developed for the cross-sectional spring-kindergarten child weights. These classes were created with 
CHAID (chi-square automatic interaction detection), using the school characteristics from the school 
nonresponse adjustments (i.e., school type, locale, region, school enrollment classified into size category). 
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and a set of child characteristics (i.e., year of birth, sex, and race/ethnicity). Data on year of birth were 
obtained from the parent interviews, while data on sex and race/ethnicity were from the child sampling 
information, which was provided by the schools. If year of birth was missing from the parent interview, 
then it was taken from the child sampling information. If sex or race/ethnicity was missing from the child 
sampling information, then they were obtained from the parent interview data. Any remaining missing 
data were imputed with the modal value from the school from which the child was sampled for this 
purpose. 



7.2. 1.2.2 Base Year Child Weights for Children Sampled in First Grade 

In spring-first grade the student sample was freshened to include first graders who had not 
been enrolled in kindergarten in 1998-99 and, therefore, had no chance of being included in the ECLS-K 
base year kindergarten sample. For this group of children who entered the sample in first grade, their 
weights need to have additional adjustments to account for the freshening procedure. See chapter 3 for a 
discussion of the student freshening in spring-first grade. 

Since each child sampled in first grade was directly linked to a child sampled in 
kindergarten, the first step was to compute a weight for the children who were sampled in kindergarten 
that reflects the school freshening subsampling and the school freshening nonresponse (some schools 
refused to provide the complete alphabetical roster of all students enrolled in first grade needed for 
freshening). This weight was then linked back to the child sampled in first grade and further adjusted for 
nonresponse due to not obtaining the data (e.g., assessment data, parent interview data) from the sample 
of freshened children. The procedures for computing the base year child weights for children sampled in 
first grade are described next. 

School Weight Adjusted for Subsampling of Schools for Freshening. The school base 
weight adjusted for base year school-level nonresponse {SCHLADW^) computed in section 7. 2. 1.1 was 
adjusted for the subsampling of schools for freshening. As noted earlier, student freshening was done in a 
50 percent subsample of schools. The school weight adjusted for school freshening subsampling, 
FRXSCHW ^ , was calculated as 



FRXSCHW^ = FRIADF^ x SCHLADW^ 



7-11 




where FRXADF^ , the adjustment factor, was computed as 



where 



FRlADFc 



^ SCHLADWi 



< ^ SCHLADWi 

i^F 

0 



if school i e F^, 
if school i € F^ 



F^ denotes the set of schools subsampled for freshening and 

F^ denotes the set of schools not subsampled for freshening. 

This adjustment was done within cells defined by school type (public, Catholic private, non- 
Catholic religious private, or nonsectarian private) and census region (Northeast, Midwest, South, or 
West). 



School Weight Adjusted for Freshening Nonresponse. The freshening procedure could 
not be applied in all designated schools because some schools did not provide the information needed for 
freshening (see chapter 3 for more details on the freshening procedures). These schools were considered 
nonrespondents. The school weight adjusted for freshening school-level nonresponse, FRISCHW^ , was 

computed as 



FRlSCHWi ^ FR2ADF ^xFRlSCHWi 



where FRIADF^ , the adjustment factor, was computed as 



FRIADF^ = 



^ FR\SCHWi X nii 
i^FER^uFENR^ 

^ FRlSCHWi X Mi 

i^FER, 



where 



w, is the original school MOS, 

FER^ denotes the set of freshening school respondents in cell c, and 
FENR^ denotes the set of freshening school nonrespondents in cell c. 
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In both the numerator and denominator of this factor, the school measure of size (i.e., the 
count of students in the school as described in section 3. 2. 2.2) was incorporated; the school measure of 
size is relevant because the weights will be used for child-level estimates, not school-level estimates. The 
nonresponse cells for this adjustment were created using school type (public. Catholic private, non- 
Catholic religious private, or nonsectarian private) and urbanicity (large city, midsize city, suburb of large 
city, suburb of midsize city, large town, small town, or rural area). 

Base Year Child Weight. Next, the school-adjusted weight was multiplied by the inverse of 
the probability of sampling the child in the base year to obtain a base year child weight for freshening. 
The base year child weight was BYCHLDW^ : 

BYCHLDWi = FRlSCHWi x (VCHLD PROBi) 
where CHLD PROB- is the within-school child selection probability. 



The base year child weight was then adjusted for base year child nonresponse because 
children who did not respond in the base year could not be linked to children in first grade in spring 2000. 
The adjusted weight, ABYCHLDW- was computed as 



ABYCHLDWi = BYADF^ x BYCHLDWi 



where BYADF^ , the adjustment factor, was computed as 



where 



^ BYCHLDWi 



BYADF^ = 



i^BY R^ ,BY_NR^ 

^ BYCHLDWi 



i^BYR^ 



BY_R c denotes the set of base year child respondents in cell c, and 
BY NR c denotes the set of base year child nonrespondents in cell c. 



The nonresponse cells were created using the school characteristics school type, locale, 
region, and school enrollment size, and the child characteristics age, sex, and race/ethnicity. 
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Base Year Child Weights Adjusted for Movers. Only children who did not move from 
their original school were designated as links to children in the freshening procedure. The children who 
moved and were followed into their new schools were not identified to participate in the freshening 
process in their new schools. As a result, all children who moved were considered nonrespondents for the 
freshening process. Additionally, nonmovers and movers who were not in first grade were not eligible for 
freshening (e.g., if the child was in kindergarten in spring 2000, he or she would be linked only to other 
kindergarten children and thus was not eligible for the freshening of first graders). An adjustment was 
necessary to account for these two groups of children and was done in two steps. 



In the first step, an adjustment was made for movers whose grade was unknown. A portion 
of the movers was assumed to be in first grade. In the second step, the weights were adjusted for children 
who were in first grade and who were not identified to participate in the freshening process because they 
moved into a new school. For this two-step adjustment, each child was classified as in table 7-2. 



Table 7-2. Groups of children defined for mover 
adjustments: School year 1999-2000 



Groups 


Mover status 


MOVl 


Mover enrolled in first grade 


MOVOTH 


Mover enrolled in another grade 


MUNK 


Mover enrolled in an unknown grade 


NM\ 


Nonmover enrolled in first grade 


NMOTH 


Nonmover enrolled in another grade 



SOURCE: U.S. Department of Education, National Center for Education 
Statistics, Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 
first grade data collection, school year 1999-2000. 



The first step adjustment for movers whose grade in spring 2000 was unknown was 



computed as 



RAMOVWli = RAMOVFl^ x ABYCHLDWi 



7-14 







where RAMOVFX^ , the adjustment factor, was computed as 



^ ABYCHLDWi 

i^MOVY, MOVOTH^ MUNK^ 

Y, ABYCHLDWi 

i^MOVY, MOVOTH^ 



if the child e MOV\, MOVOTH in cell c. 



RAMOVFl 



if the child e MUNK in cell c, 



1 



if the child e NMl, NMOTH in cell c. 



The second step adjustment for movers who could not be used as links for freshening was 

computed as 



RAMOVWl^ = RAMOVFl^ x RAMOVW\ 
where RAMOVFl^ , the adjustment factor, was computed as 



Y RAMOVWli 
i&NMY,MOVY 

Y RAMOVWli 
isNMY 



if the child e NM\ in cell c. 



RAMOVF2 



if the child eMOVl in cell c. 



1 



if the child g NMOTH, MOVOTH in cell c. 



This two-step adjustment was done within cells defined by school type and census region. 

The weights thus created for children sampled in kindergarten were then linked to the 
children that they brought into the sample in first grade through sample freshening. In other words, the 
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weight of the child sampled in first grade was defined at this point to be the weight computed for the child 
sampled in kindergarten that was responsible for bringing the first grader into the sample. 

For the next step in the computation of the spring-third grade child weights, the two groups 
of children — base year respondents and children sampled in first grade through sample freshening — were 
put together, and a common variable and label were used to designate the initial child weight. This is the 
base year child weight as computed above for each group of children: 

f ABYCHLD W; if base year respondent, 

ICHLDWi^i 

\^R4MOVW2i if sampled in first grade 

The initial child weights ICHLDW were adjusted for movers between the base year and 
third grade and for nonresponse in third grade, and raked to sampled-based control totals to obtain the 
final spring-third grade child weights. These adjustments and raking procedures are described below. 



7.2.2 Adjustment for Movers Between the Base Year and Third Grade 

In the ECLS-K, a child could move more than once and at different times. For example, a 
child could move out of his or her original sample school because the school did not have grades higher 
than kindergarten. Then he could move again between first and third grade. Once a child was identified as 
a mover, he stayed a mover (unless he or she moved back to the original sample school). For example, a 
child who moved between kindergarten and first grade, but stayed in the same school from second and 
third grade would be considered a mover for the third grade. The spring-first grade follow flags indicating 
that a child should be followed if he moved were maintained for all children in the spring-third grade 
sample except for children whose home language was not English. For these language minority children, 
their spring-first grade follow flags were switched to 1 (i.e., to be followed if moved) if they were not 
already equal to 1 and if they had not already been subsampled out because they were identified as 
movers in spring-first grade.^ Thus, children who moved out of their original sample school were 
followed in the random 50 percent of schools where the follow flag was set to 1, and language minority 



^In order to maximize the number of children with longitudinal data, care was taken during spring-first grade sampling to ensure that any child 
who had been flagged to be followed in fall-first grade continued to be followed in spring-first grade. In third grade, these children continued to 
be followed. 
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children were followed at 100 percent if they had not moved previously.*’ The child weight adjusted for 
subsampling movers R5CHLDW\i was computed as 

R5CHLDW\i = RSADFl^ x ICHLDW^ 

where R5ADF\^ , the adjustment factor, was computed as 



fl 



if the child in cell c is not a mover, 



R5ADF\^ 



Y, ICHLDWi 

ieMOVER^ with 
FOLLOW_F in(Q,\) 

Y ICHLDWi 

i eMOVFF. with 
FOLLOW_F=l 



if the child is a mover in cell c and FOLLOW F = 1, 



[0 



if the child is a mover in cell c and FOLLOW F = 0. 



For the cross-sectional weights, the mover adjustment factor was computed within cells 
created using the following characteristics: whether children were sampled in kindergarten or first grade, 
whether they were movers in spring-first grade, whether they were language minority children, the school 
type of their original sample school, and the region where their original sample school was located. For 
the longitudinal weights, a longitudinal mover follow status was created. This status took into account 
whether the child moved from his or her original school in fall-first grade, spring-first grade, or spring- 
third grade (for longitudinal weights involving the fall-first grade data) or whether the child moved from 
his or her original school in spring-first grade or spring-third grade (for the other longitudinal weights). If 
a child moved in either round, he or she was considered a mover. Adjustment cells were created in the 
same way as for cross-sectional weights. However, for longitudinal weights involving all five rounds of 
data, only mover status and language minority status were used to create adjustment cells. Appendix 7A 
gives the cell definitions for the mover adjustment for cross-sectional and longitudinal weights. 



A few children with large cross-sectional child weights (C5CW0) and a few children with 
large spring-first grade/spring-third grade longitudinal child weights (C45CW0) had their weights 



^ Language minority children who moved between kindergarten and first grade were followed to their first grade new schools at a rate of 
approximately 50 percent. If these movers moved again between first grade and third grade, then they were followed with certainty. The only 
language minority children who left the sample are those who moved between kindergarten and first grade and were not followed into their first 
grade schools. 
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trimmed by half. However, the weights were not redistributed because the total sum of weights was re- 
established in the raking procedure that came later. 



7.2.3 Adjustment for Third Grade Nonresponse 



After the adjustment for subsampling movers, the child weights were adjusted for 
nonresponse. As in spring-first grade, the nonresponse adjustment was done in two steps. In the first step, 
the adjustment was for children whose eligibility was not determined (unknown eligibility). A portion of 
children of unknown eligibility was assumed to be ineligible. In the second step, the adjustment was for 
eligible nonrespondents. To carry out these adjustments, each child was classified as in table 7-3. 



Table 7-3. Groups of children defined for nonresponse adjustments: 
School year 2001-02 



Groups 


Eligibility and response status 


ER 


Eligible respondent 


ENR 


Eligible nonrespondent 


IN 


Ineligible (out of the country or deceased) 


UNK 


Unknown eligibility (mover who could not be located) 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Early 
Childhood Longitudinal Study, Kindergarten Class of 1998-99 third grade data collection, school 
year 2001-02. 



The child weight adjusted for nonrespondents with unknown eligibility, RSCHLDWli , was 

computed as 



RSCHLDWli^ R5ADF2^xR5CHLDW\ 
where R5ADF2 ^ , the adjustment factor, was computed as 



R5ADF2^ 



^ R5CHLDW\i 

ieER^, ENR^ IN^,UNK^ 

^ R5CHLDW\i 

ieERc,ENRcJNc 



if the child g ER, ENR, IN in cell c. 



[O if the child g UNK in cell c. 

In the second adjustment, the child weight adjusted for eligible nonrespondents, 
RSCHLDW2>i , was computed as 
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R5CHLDW3i = R5ADF3^ x RSCHLDWli 
where R5ADF3^ , the adjustment factor, was computed as 



^ RSCHLDWli 

ieER^,ENR^ 

^ R5CHLDW2i 

ieER^ 



if the child e ER in cell c. 



R5ADF3 



if the child e ENR in cell c. 



1 



if the child e IN in cell c. 



In both steps of the adjustment, separate nonresponse classes were created for movers and 
nonmovers using various combinations of response status of child assessments and/or parent interviews in 
the base year^ as well as whether children belong to the language minority group, the type of family (i.e., 
one-parent family, two-parent family, or other type of family) as identified during the spring-third grade 
parent interviews (C5PW0 only), and the school type including whether the child was homeschooled 
(C5CPTW0 only). These cells vary according to each type of weights as shown in appendix 7B. 



7.2.4 Raking to Sample-Based Control Totals 

To reduce the variability due to the subsampling of schools and movers, the child weights 
were then raked (i.e., calibrated) to sample-based control totals computed using the initial child weights 
described in section 7.2.1. A file was created with the initial child weights and school and child 
characteristics collected in the base year or first grade year (such as school type, region, urbanicity, sex, 
age, race/ethnicity, SES, language minority status, whether sampled in kindergarten or first grade, and if 
sampled in kindergarten, mover status in spring-first grade) to be used in the computation of the control 
totals. The child records included in this file are records of base year respondents and records of eligible 

Tor example, a dichotomous variable was created to indicate whether the child has positive CICWO and C2CW0 and CIPWO and C2PW0 
weights; this variable was then used to create nonresponse classes. 
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children sampled in first grade, including records of children who became ineligible in spring-third grade. 
The sum of the initial weights thus calculated is the estimated number of children who were in 
kindergarten in 1998-99 or first grade in 1999-2000. In the previous steps, the weights of the 
nonresponding children were distributed to the responding children while the weights of the ineligible 
children were not affected by this weighting step. The weights of the ineligible children were set to zero 
at the end of this process because these children are not meant to be included in the analysis of the spring- 
third grade data. The reason for including the ineligible children in the raking step is that these children 
were included in the sampled-based control totals. 

Before raking the C5CPT weights, a small number of responding movers had their 
nonresponse-adjusted weights trimmed and the excess weight redistributed among the remaining 
responding movers so that the sum of weights before trimming was equal to the sum of weights after 
trimming. 



The raked child weight or spring-third grade final child weight, R5CHLDW , was 

calculated as 



R5CHLDW4i = R5ADF4^xR5CHLDW3i 



where R5ADF4^ , the raking factor, was computed as 



R5ADF4^ 



SMPCNT^ 

^R5CHLDW3i 

iGC 



where SMP _CNT^ is the sample-based control total for raking cell c. Weights of children who become 
ineligible in spring-third grade were set to zero after this step. 

This raking procedure is essentially a multivariate poststratification. Raking cells (also 
known as raking dimensions because they typically involve more than one variable, for example, gender 
by age) were created using school and child characteristics collected in the base year or first grade year: 
school type, region, urbanicity, sex, age, race/ethnicity, SES, language minority status, whether sampled 
in kindergarten or first grade, and if sampled in kindergarten, mover status in spring-first grade. Appendix 
7C gives the raking dimensions used for spring-third grade. 
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There was no restriction set in the number of iterations during the raking procedure. The 
procedure was allowed to run until complete convergence was achieved within a control total. Typically, 
this occurred with 12 to 16 iterations. 



7.2.5 Replicate Weights 

For each set of cross-sectional and longitudinal weights included in the third grade data fde, 
a set of replicate weights was computed. All adjustments to the full sample weights were repeated for the 
replicate weights. The replication scheme used for the base year was used for all of the spring-third grade 
weights that did not contain any fall-first grade component. If a fall-first grade component was included 
in the definition of the respondents for the weight, then the replication scheme used for fall-first grade 
estimates was used. 

Replicate weights are needed to estimate the standard errors of survey estimates. A total of 
90 replicate weights were computed using the paired jackknife method (denoted as JK2) for the spring- 
third grade weights if no fall-first grade component was included. These replicates take into account the 
Durbin method of PSU selection. A total of 40 replicates using the paired jackknife method were created 
for the weights that contain a fall-first grade component. The smaller number of replicates is due to the 
fact that only 30 percent of the full sample of schools was included in the fall-first grade subsample. Only 
one of the two sampled PSUs in the non-self-representing strata was kept in the sample. Consequently, 
the fall-first grade weights do not account for the Durbin PSU sampling method, which required two 
PSUs per stratum. 

The procedures used to compute the replicate weights took into account each step of the 
weighting process. One feature that is somewhat uncommon in practice is the use of sample-based raking 
as described in section 7.2.4. The control totals ( SMP _CNT^) used for raking are estimates calculated 
using the initial child weights (ICHLDWi). When population-based raking is used, these totals are 
assumed to be numbers that are known and without sampling error. To reflect the variability of the control 
totals in the sample-based raking, a set of replicate control totals was calculated rather than having a 
constant set of totals. Each replicate weight was then raked to the corresponding replicate-based control 
total. The result of this process was that each replicate retained the variability associated with the original 
sample estimates of the control totals. As with the full sample weight, the raking procedure was allowed 
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to run until complete convergence. For spring-third grade, full convergence was achieved after 12 to 16 
iterations for each replicate weight. 



7.2.5.1 Replicate Weights for Samples not Involving Fall-First Grade 

For the original ECLS-K design in the base year, replicate weights were created taking into 
account the Durbin method of PSU selection. The Durbin method selects two first-stage units per stratum 
without replacement, with probability proportional to size and a known joint probability of inclusion. 

In the ECLS-K PSU sample design, there were 24 self-representing (SR) strata and 38 non- 
self-representing (NSR) strata. Among the 38 NSR strata, 11 strata were identified as Durbin strata and 
were treated as SR strata for variance estimation. The purpose of the Durbin strata is to allow variances to 
be estimated as if the first-stage units were selected with replacement. This brings the number of SR PSUs 
to 46 (24 original SR PSUs and 22 Durbin PSUs from the 1 1 Durbin strata). The remaining 54 NSR PSUs 
are in 27 NSR strata; thus 27 replicates were formed, each corresponding to one NSR stratum. For the SR 
strata, 63 replicates were formed. The 90 replicates will yield about 76 degrees of freedom for calculating 
confidence intervals for many survey estimates. 

As stated earlier, the sample of PSUs was divided into 90 replicates or variance strata. The 
27 NSR strata formed 27 variance strata of two PSUs each; each PSU formed a variance unit within a 
variance stratum. All schools within an NSR PSU were assigned to the same variance unit and variance 
stratum. Sampled schools in the 46 SR PSUs were grouped into 63 variance strata. In the SR PSUs, 
schools were directly sampled and constituted PSUs. Public schools were sampled from within PSU while 
private schools were pooled into one sampling stratum and selected systematically (except in the SR 
PSUs identified through the Durbin method where private schools were treated as if they were sampled 
from within PSU). Schools were sorted by sampling stratum, type of school (from the original sample or 
newly selected as part of freshening), type of frame (for new schools only), and their original order of 
selection (within stratum). From this sorted list, they were grouped into pairs within each sampling 
stratum; the last pair in the stratum may be a triplet if the number of schools in the stratum is odd. This 
operation resulted in a number of ordered preliminary variance strata of two or three units each. The first 
ordered 63 strata were then numbered sequentially from 1 to 63; the next ordered 63 strata were also 
numbered sequentially from 1 to 63, and so on until the list was exhausted, thus forming the desired 63 
variance strata. 
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In strata with two units, a unit being a PSU in the case of NSR PSUs and a school in the case 
of SR PSUs, the base weight of the first unit was doubled to form the replicate weight, while the base 
weight of the second unit was multiplied by zero. In strata with three units, two variance strata were 
created: in the first variance stratum, the base weight of two of the three units was multiplied by 1.5 to 
form the replicate weight, and the base weight of the last unit was multiplied by zero; in the second 
variance stratum, the base weight of a different group of two units was multiplied by 1.5, and the base 
weight of the third unit was multiplied by zero. Multiplying the base weight in a unit by zero is equivalent 
to dropping one unit as required by the jackknife method. All adjustments to the full sample weights were 
repeated for the replicate weights. For each full sample weight, there are 90 replicate weights with the 
same weight prefix. 

A child sampled in first grade through the freshening process was assigned to the same 
replicate as the originally sampled child to whom the child was linked. When the child sampled in first 
grade was assigned a full sample weight, he or she was assigned the replicate weights in the same manner. 



7.2.S.2 Replicate Weights for Samples Involving Fall-First Grade 

For the two longitudinal weights involving fall-first grade (C15SC0 and C15SP0), there 
are 40 replicate weights. The smaller number of replicates was due to the fact that only a subsample of 
schools was included in the fall- first grade sample. The weights associated with the fall-first grade data do 
not account for the Durbin method of selecting PSUs, since it no longer applied. Rather, they reflect the 
fact that only one of the two sampled PSUs in the NSR strata was kept in the subsample. To account for 
this feature, pairs of similar NSR PSUs were collapsed into 19 variance strata. The SR PSUs account for 
the remaining 2 1 variance strata. 



7.3 Variance Estimation 

7.3.1 Jackknife Method 

The final full sample and the adjusted replicate weights can be used to compute estimates of 
variance for survey estimates using WesVar, AM, or other software that handles replicate weights. The 
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estimate of variance is the sum of the squared deviations of the replicate estimates from the full-sample 
estimate: 

g=i 

where 

6 is the population statistic of interest, 

9 is the estimate of 6 based on the full sample, 

G is the number of replicates, and 

9^g^ is the g* replicate estimate of 0 based on the observations included in the g* 
replicate. 



7.3.2 Taylor Series Method 

Variance stratum and variance unit (first-stage sample unit) identifiers were also created to 
be used in statistical software that computes variance estimates based on the Taylor series method (e.g., 
SUDAAN, Stata, SAS, and AM). In this method, a linear approximation to a statistic is formed and then 
substituted into the formula for calculating the variance of a linear estimate appropriate for the sample 
design. 



If 7 = denotes a p-dimensional vector of population parameters, Y = is 

the corresponding vector of estimators based on a sample s of size n{s), 0 = g{Y ) is the population 
parameter of interest, and 0 = g(7)is an estimator of 0 , then 



0-0. X 



7.-7.) 



and 



v0 .V 



V i=i <^ 7 , y j=\ 1=1 j YVi 
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The Taylor series method relies on a simplified procedure for estimating the variance for a 
linear statistic even with a complex sample design and is valid in large samples in which the first stage 
units are sampled with replacement. For the ECLS-K, this simplified method does not capture the 
variance related to the Durbin sampling method, the effects of the adjustments of the weights for 
nonresponse, or the sample-based raking procedures. These effects are not captured in the Taylor series 
variance estimates mainly because each adjustment corresponds to a different estimator that the variance 
estimation software does not support. In some cases these adjustments may have only a minor effect on 
the variance estimates, but in other cases the effects could be more substantial. 

For software that uses the Taylor series method, the variance strata and PSUs must be 
defined. For the spring-third grade ECLS-K, the Taylor variance strata were assigned by sequentially 
numbering the sampling strata and collapsing any stratum with one PSU with an adjacent stratum. In 
theory, any variance stratum with fewer than two responding units would be combined with an adjacent 
stratum, but this did not happen in the ECLS-K. The variance units were assigned by sequentially 
numbering the first-stage sampling units within sampling strata. For example, for C5CW0, Taylor 
variance strata were numbered sequentially from 1 to 90. Within each Taylor stratum, Taylor units were 
numbered sequentially from 1 to the total number of units in the stratum. This procedure was done 
separately for each cross-sectional and longitudinal weight. 
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